agora inbox for pgsql-hackers@postgresql.org
help / color / mirror / Atom feedSpeed up COPY FROM text/CSV parsing using SIMD
178+ messages / 11 participants
[nested] [flat]
* Speed up COPY FROM text/CSV parsing using SIMD
@ 2025-08-07 01:48 Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
0 siblings, 1 reply; 178+ messages in thread
From: Shinya Kato @ 2025-08-07 01:48 UTC (permalink / raw)
To: pgsql-hackers
Hi hackers,
I have implemented SIMD optimization for the COPY FROM (FORMAT {csv,
text}) command and observed approximately a 5% performance
improvement. Please see the detailed test results below.
Idea
====
The current text/CSV parser processes input byte-by-byte, checking
whether each byte is a special character (\n, \r, quote, escape) or a
regular character, and transitions states in a state machine. This
sequential processing is inefficient and likely causes frequent branch
mispredictions due to the many if statements.
I thought this problem could be addressed by leveraging SIMD and
vectorized operations for faster processing.
Implementation Overview
=======================
1. Create a vector of special characters (e.g., Vector8 nl =
vector8_broadcast('\n');).
2. Load the input buffer into a Vector8 variable called chunk.
3. Perform vectorized operations between chunk and the special
character vectors to check if the buffer contains any special
characters.
4-1. If no special characters are found, advance the input_buf_ptr by
sizeof(Vector8).
4-2. If special characters are found, advance the input_buf_ptr as far
as possible, then fall back to the original text/CSV parser for
byte-by-byte processing.
Test
====
I tested the performance by measuring the time it takes to load a CSV
file created using the attached SQL script with the following COPY
command:
=# COPY t FROM '/tmp/t.csv' (FORMAT csv);
Environment
-----------
OS: Rocky Linux 9.6
CPU: Intel Core i7-10710U (6 Cores / 12 Threads, 1.1 GHz Base / 4.7
GHz Boost, AVX2 & FMA supported)
Time
----
master: 02.44.943
patch applied: 02:36.878 (about 5% faster)
Perf
----
Each call graphs are attached and the rates of CopyReadLineText are:
master: 12.15%
patch applied: 8.04%
Thought?
I would appreciate feedback on the implementation and any suggestions
for further improvement.
--
Best regards,
Shinya Kato
NTT OSS Center
Attachments:
[application/octet-stream] v1-0001-Speed-up-COPY-FROM-text-CSV-parsing-using-SIMD.patch (3.9K, ../../CAOzEurSW8cNr6TPKsjrstnPfhf4QyQqB4tnPXGGe8N4e_v7Jig@mail.gmail.com/2-v1-0001-Speed-up-COPY-FROM-text-CSV-parsing-using-SIMD.patch)
download | inline diff:
From 5ae3be7d262e4251bf21ac0c73b3e0ebc2ba615d Mon Sep 17 00:00:00 2001
From: Shinya Kato <shinya11.kato@gmail.com>
Date: Mon, 28 Jul 2025 22:08:20 +0900
Subject: [PATCH v1] Speed up COPY FROM text/CSV parsing using SIMD
The inner loop of CopyReadLineText scans for newlines and other special
characters by processing the input byte-by-byte. For large inputs, this
can be a performance bottleneck.
This commit introduces a SIMD-accelerated path. When not parsing inside
a quoted field, we can use vector instructions to scan the input buffer
for any character of interest in 16-byte chunks. This significantly
improves performance, especially for data with long, unquoted fields.
---
src/backend/commands/copyfromparse.c | 72 ++++++++++++++++++++++++++++
1 file changed, 72 insertions(+)
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index b1ae97b833d..5aba0fa6cb7 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -71,7 +71,9 @@
#include "mb/pg_wchar.h"
#include "miscadmin.h"
#include "pgstat.h"
+#include "port/pg_bitutils.h"
#include "port/pg_bswap.h"
+#include "port/simd.h"
#include "utils/builtins.h"
#include "utils/rel.h"
@@ -1255,6 +1257,14 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
char quotec = '\0';
char escapec = '\0';
+#ifndef USE_NO_SIMD
+ Vector8 nl = vector8_broadcast('\n');
+ Vector8 cr = vector8_broadcast('\r');
+ Vector8 bs = vector8_broadcast('\\');
+ Vector8 quote;
+ Vector8 escape;
+#endif
+
if (is_csv)
{
quotec = cstate->opts.quote[0];
@@ -1262,6 +1272,12 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
/* ignore special escape processing if it's the same as quotec */
if (quotec == escapec)
escapec = '\0';
+
+#ifndef USE_NO_SIMD
+ quote = vector8_broadcast(quotec);
+ if (quotec != escapec)
+ escape = vector8_broadcast(escapec);
+#endif
}
/*
@@ -1328,6 +1344,62 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
need_data = false;
}
+#ifndef USE_NO_SIMD
+ /*
+ * SIMD instructions are used here to efficiently scan the input buffer
+ * for special characters (e.g., newline, carriage return, quotes, or
+ * escape characters). This approach significantly improves performance
+ * compared to byte-by-byte iteration, especially for large input
+ * buffers.
+ *
+ * However, SIMD optimization cannot be applied in the following cases:
+ * - Inside quoted fields, where escape sequences and closing quotes
+ * require sequential processing to handle correctly.
+ * - When the remaining buffer size is smaller than the size of a SIMD
+ * vector register, as SIMD operations require processing data in
+ * fixed-size chunks.
+ */
+ if (!in_quote && copy_buf_len - input_buf_ptr >= sizeof(Vector8))
+ {
+ Vector8 chunk;
+ Vector8 match;
+ uint32 mask;
+
+ /* Load a chunk of data into a vector register */
+ vector8_load(&chunk, (const uint8 *) ©_input_buf[input_buf_ptr]);
+
+ /* Create a mask of all special characters we need to stop at */
+ match = vector8_or(vector8_eq(chunk, nl), vector8_eq(chunk, cr));
+
+ if (is_csv)
+ {
+ match = vector8_or(match, vector8_eq(chunk, quote));
+ if (escapec != '\0')
+ match = vector8_or(match, vector8_eq(chunk, escape));
+ }
+ else
+ match = vector8_or(match, vector8_eq(chunk, bs));
+
+ /* Check if we found any special characters */
+ mask = vector8_highbit_mask(match);
+ if (mask != 0)
+ {
+ /*
+ * Found a special character. Advance up to that point and let
+ * the scalar code handle it.
+ */
+ int advance = pg_rightmost_one_pos32(mask);
+ input_buf_ptr += advance;
+ }
+ else
+ {
+ /* No special characters found, so skip the entire chunk */
+ input_buf_ptr += sizeof(Vector8);
+ continue;
+ }
+ }
+#endif
+
/* OK to fetch a character */
prev_raw_ptr = input_buf_ptr;
c = copy_input_buf[input_buf_ptr++];
--
2.47.1
[application/octet-stream] test.sql (1.5K, ../../CAOzEurSW8cNr6TPKsjrstnPfhf4QyQqB4tnPXGGe8N4e_v7Jig@mail.gmail.com/3-test.sql)
download
[image/svg+xml] master.svg (332.6K, ../../CAOzEurSW8cNr6TPKsjrstnPfhf4QyQqB4tnPXGGe8N4e_v7Jig@mail.gmail.com/4-master.svg)
download | view image
[image/svg+xml] patch_applied.svg (343.1K, ../../CAOzEurSW8cNr6TPKsjrstnPfhf4QyQqB4tnPXGGe8N4e_v7Jig@mail.gmail.com/5-patch_applied.svg)
download | view image
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
@ 2025-08-07 11:15 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 09:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Ants Aasma <ants.aasma@cybertec.at>
0 siblings, 2 replies; 178+ messages in thread
From: Nazir Bilal Yavuz @ 2025-08-07 11:15 UTC (permalink / raw)
To: Shinya Kato <shinya11.kato@gmail.com>; +Cc: pgsql-hackers
Hi,
Thank you for working on this!
On Thu, 7 Aug 2025 at 04:49, Shinya Kato <shinya11.kato@gmail.com> wrote:
>
> Hi hackers,
>
> I have implemented SIMD optimization for the COPY FROM (FORMAT {csv,
> text}) command and observed approximately a 5% performance
> improvement. Please see the detailed test results below.
I have been working on the same idea. I was not moving input_buf_ptr
as far as possible, so I think your approach is better.
Also, I did a benchmark on text format. I created a benchmark for line
length in a table being from 1 byte to 1 megabyte.The peak improvement
is line length being 4096 and the improvement is more than 20% [1], I
saw no regression on your patch.
> Idea
> ====
> The current text/CSV parser processes input byte-by-byte, checking
> whether each byte is a special character (\n, \r, quote, escape) or a
> regular character, and transitions states in a state machine. This
> sequential processing is inefficient and likely causes frequent branch
> mispredictions due to the many if statements.
>
> I thought this problem could be addressed by leveraging SIMD and
> vectorized operations for faster processing.
>
> Implementation Overview
> =======================
> 1. Create a vector of special characters (e.g., Vector8 nl =
> vector8_broadcast('\n');).
> 2. Load the input buffer into a Vector8 variable called chunk.
> 3. Perform vectorized operations between chunk and the special
> character vectors to check if the buffer contains any special
> characters.
> 4-1. If no special characters are found, advance the input_buf_ptr by
> sizeof(Vector8).
> 4-2. If special characters are found, advance the input_buf_ptr as far
> as possible, then fall back to the original text/CSV parser for
> byte-by-byte processing.
>
...
> Thought?
> I would appreciate feedback on the implementation and any suggestions
> for further improvement.
I have a couple of ideas that I was working on:
---
+ * However, SIMD optimization cannot be applied in the following cases:
+ * - Inside quoted fields, where escape sequences and closing quotes
+ * require sequential processing to handle correctly.
I think you can continue SIMD inside quoted fields. Only important
thing is you need to set last_was_esc to false when SIMD skipped the
chunk.
---
+ * - When the remaining buffer size is smaller than the size of a SIMD
+ * vector register, as SIMD operations require processing data in
+ * fixed-size chunks.
You run SIMD when 'copy_buf_len - input_buf_ptr >= sizeof(Vector8)'
but you only call CopyLoadInputBuf() when 'input_buf_ptr >=
copy_buf_len || need_data' so basically you need to wait at least the
sizeof(Vector8) character to pass for the next SIMD. And in the worst
case; if CopyLoadInputBuf() puts one character less than
sizeof(Vector8), then you can't ever run SIMD. I think we need to make
sure that CopyLoadInputBuf() loads at least the sizeof(Vector8)
character to the input_buf so we do not encounter that problem.
---
What do you think about adding SIMD to CopyReadAttributesText() and
CopyReadAttributesCSV() functions? When I add your SIMD approach to
CopyReadAttributesText() function, the improvement on the 4096 byte
line length input [1] goes from 20% to 30%.
---
I shared my ideas as a Feedback.txt file (.txt to stay off CFBot's
radar for this thread). I hope these help, please let me know if you
have any questions.
--
Regards,
Nazir Bilal Yavuz
Microsoft
From b13f4cdf134eef5fbecf9ea06f9b1c99890b7c02 Mon Sep 17 00:00:00 2001
From: Nazir Bilal Yavuz <byavuz81@gmail.com>
Date: Thu, 7 Aug 2025 13:27:34 +0300
Subject: [PATCH] Feedback
---
src/backend/commands/copyfromparse.c | 55 ++++++++++++++++++++++++++--
1 file changed, 51 insertions(+), 4 deletions(-)
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index 5aba0fa6cb7..dae5c1f698c 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -670,8 +670,12 @@ CopyLoadInputBuf(CopyFromState cstate)
/* If we now have some unconverted data, try to convert it */
CopyConvertBuf(cstate);
- /* If we now have some more input bytes ready, return them */
- if (INPUT_BUF_BYTES(cstate) > nbytes)
+ /*
+ * If we now have at least sizeof(Vector8) input bytes ready, return
+ * them. This is beneficial for SIMD processing in the
+ * CopyReadLineText() function.
+ */
+ if (INPUT_BUF_BYTES(cstate) > nbytes + sizeof(Vector8))
return;
/*
@@ -1322,7 +1326,7 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
* unsafe with the old v2 COPY protocol, but we don't support that
* anymore.
*/
- if (input_buf_ptr >= copy_buf_len || need_data)
+ if (input_buf_ptr + sizeof(Vector8) >= copy_buf_len || need_data)
{
REFILL_LINEBUF;
@@ -1359,7 +1363,7 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
* vector register, as SIMD operations require processing data in
* fixed-size chunks.
*/
- if (!in_quote && copy_buf_len - input_buf_ptr >= sizeof(Vector8))
+ if (copy_buf_len - input_buf_ptr >= sizeof(Vector8))
{
Vector8 chunk;
Vector8 match;
@@ -1395,6 +1399,7 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
{
/* No special characters found, so skip the entire chunk */
input_buf_ptr += sizeof(Vector8);
+ last_was_esc = false;
continue;
}
}
@@ -1650,6 +1655,11 @@ CopyReadAttributesText(CopyFromState cstate)
char *cur_ptr;
char *line_end_ptr;
+#ifndef USE_NO_SIMD
+ Vector8 bs = vector8_broadcast('\\');
+ Vector8 delim = vector8_broadcast(delimc);;
+#endif
+
/*
* We need a special case for zero-column tables: check that the input
* line is empty, and return.
@@ -1717,6 +1727,43 @@ CopyReadAttributesText(CopyFromState cstate)
{
char c;
+#ifndef USE_NO_SIMD
+ if (line_end_ptr - cur_ptr >= sizeof(Vector8))
+ {
+ Vector8 chunk;
+ Vector8 match;
+ uint32 mask;
+
+ /* Load a chunk of data into a vector register */
+ vector8_load(&chunk, (const uint8 *) cur_ptr);
+
+ /* Create a mask of all special characters we need to stop at */
+ match = vector8_or(vector8_eq(chunk, bs), vector8_eq(chunk, delim));
+
+ /* Check if we found any special characters */
+ mask = vector8_highbit_mask(match);
+ if (mask != 0)
+ {
+ /*
+ * Found a special character. Advance up to that point and let
+ * the scalar code handle it.
+ */
+ int advance = pg_rightmost_one_pos32(mask);
+ memcpy(output_ptr, cur_ptr, advance);
+ output_ptr += advance;
+ cur_ptr += advance;
+ }
+ else
+ {
+ /* No special characters found, so skip the entire chunk */
+ memcpy(output_ptr, cur_ptr, sizeof(Vector8));
+ output_ptr += sizeof(Vector8);
+ cur_ptr += sizeof(Vector8);
+ continue;
+ }
+ }
+#endif
+
end_ptr = cur_ptr;
if (cur_ptr >= line_end_ptr)
break;
--
2.50.1
Attachments:
[text/plain] Feedback.txt (3.4K, ../../CAN55FZ247JdiT8Sd1SRiyOJxk3Ei=pDCL4kpdP=HqLRjOhKf1Q@mail.gmail.com/2-Feedback.txt)
download | inline diff:
From b13f4cdf134eef5fbecf9ea06f9b1c99890b7c02 Mon Sep 17 00:00:00 2001
From: Nazir Bilal Yavuz <byavuz81@gmail.com>
Date: Thu, 7 Aug 2025 13:27:34 +0300
Subject: [PATCH] Feedback
---
src/backend/commands/copyfromparse.c | 55 ++++++++++++++++++++++++++--
1 file changed, 51 insertions(+), 4 deletions(-)
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index 5aba0fa6cb7..dae5c1f698c 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -670,8 +670,12 @@ CopyLoadInputBuf(CopyFromState cstate)
/* If we now have some unconverted data, try to convert it */
CopyConvertBuf(cstate);
- /* If we now have some more input bytes ready, return them */
- if (INPUT_BUF_BYTES(cstate) > nbytes)
+ /*
+ * If we now have at least sizeof(Vector8) input bytes ready, return
+ * them. This is beneficial for SIMD processing in the
+ * CopyReadLineText() function.
+ */
+ if (INPUT_BUF_BYTES(cstate) > nbytes + sizeof(Vector8))
return;
/*
@@ -1322,7 +1326,7 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
* unsafe with the old v2 COPY protocol, but we don't support that
* anymore.
*/
- if (input_buf_ptr >= copy_buf_len || need_data)
+ if (input_buf_ptr + sizeof(Vector8) >= copy_buf_len || need_data)
{
REFILL_LINEBUF;
@@ -1359,7 +1363,7 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
* vector register, as SIMD operations require processing data in
* fixed-size chunks.
*/
- if (!in_quote && copy_buf_len - input_buf_ptr >= sizeof(Vector8))
+ if (copy_buf_len - input_buf_ptr >= sizeof(Vector8))
{
Vector8 chunk;
Vector8 match;
@@ -1395,6 +1399,7 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
{
/* No special characters found, so skip the entire chunk */
input_buf_ptr += sizeof(Vector8);
+ last_was_esc = false;
continue;
}
}
@@ -1650,6 +1655,11 @@ CopyReadAttributesText(CopyFromState cstate)
char *cur_ptr;
char *line_end_ptr;
+#ifndef USE_NO_SIMD
+ Vector8 bs = vector8_broadcast('\\');
+ Vector8 delim = vector8_broadcast(delimc);;
+#endif
+
/*
* We need a special case for zero-column tables: check that the input
* line is empty, and return.
@@ -1717,6 +1727,43 @@ CopyReadAttributesText(CopyFromState cstate)
{
char c;
+#ifndef USE_NO_SIMD
+ if (line_end_ptr - cur_ptr >= sizeof(Vector8))
+ {
+ Vector8 chunk;
+ Vector8 match;
+ uint32 mask;
+
+ /* Load a chunk of data into a vector register */
+ vector8_load(&chunk, (const uint8 *) cur_ptr);
+
+ /* Create a mask of all special characters we need to stop at */
+ match = vector8_or(vector8_eq(chunk, bs), vector8_eq(chunk, delim));
+
+ /* Check if we found any special characters */
+ mask = vector8_highbit_mask(match);
+ if (mask != 0)
+ {
+ /*
+ * Found a special character. Advance up to that point and let
+ * the scalar code handle it.
+ */
+ int advance = pg_rightmost_one_pos32(mask);
+ memcpy(output_ptr, cur_ptr, advance);
+ output_ptr += advance;
+ cur_ptr += advance;
+ }
+ else
+ {
+ /* No special characters found, so skip the entire chunk */
+ memcpy(output_ptr, cur_ptr, sizeof(Vector8));
+ output_ptr += sizeof(Vector8);
+ cur_ptr += sizeof(Vector8);
+ continue;
+ }
+ }
+#endif
+
end_ptr = cur_ptr;
if (cur_ptr >= line_end_ptr)
break;
--
2.50.1
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
@ 2025-08-11 08:52 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
1 sibling, 1 reply; 178+ messages in thread
From: Nazir Bilal Yavuz @ 2025-08-11 08:52 UTC (permalink / raw)
To: Shinya Kato <shinya11.kato@gmail.com>; +Cc: pgsql-hackers
Hi,
On Thu, 7 Aug 2025 at 14:15, Nazir Bilal Yavuz <byavuz81@gmail.com> wrote:
>
> On Thu, 7 Aug 2025 at 04:49, Shinya Kato <shinya11.kato@gmail.com> wrote:
> >
> > I have implemented SIMD optimization for the COPY FROM (FORMAT {csv,
> > text}) command and observed approximately a 5% performance
> > improvement. Please see the detailed test results below.
>
> Also, I did a benchmark on text format. I created a benchmark for line
> length in a table being from 1 byte to 1 megabyte.The peak improvement
> is line length being 4096 and the improvement is more than 20% [1], I
> saw no regression on your patch.
I did the same benchmark for the CSV format. The peak improvement is
line length being 4096 and the improvement is more than 25% [1]. I saw
a 5% regression on the 1 byte benchmark, there are no other
regressions.
> What do you think about adding SIMD to CopyReadAttributesText() and
> CopyReadAttributesCSV() functions? When I add your SIMD approach to
> CopyReadAttributesText() function, the improvement on the 4096 byte
> line length input [1] goes from 20% to 30%.
I wanted to try using SIMD in CopyReadAttributesCSV() as well. The
improvement on the 4096 byte line length input [1] goes from 25% to
35%, the regression on the 1 byte input is the same.
CopyReadAttributesCSV() changes are attached as feedback v2.
--
Regards,
Nazir Bilal Yavuz
Microsoft
From 203d648c4cf64c6d629f2abc719a371dd0393e22 Mon Sep 17 00:00:00 2001
From: Nazir Bilal Yavuz <byavuz81@gmail.com>
Date: Thu, 7 Aug 2025 13:27:34 +0300
Subject: [PATCH v2] Feedback
---
src/backend/commands/copyfromparse.c | 176 ++++++++++++++++++++++++---
1 file changed, 160 insertions(+), 16 deletions(-)
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index 5aba0fa6cb7..7b83e64e23b 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -670,8 +670,12 @@ CopyLoadInputBuf(CopyFromState cstate)
/* If we now have some unconverted data, try to convert it */
CopyConvertBuf(cstate);
- /* If we now have some more input bytes ready, return them */
- if (INPUT_BUF_BYTES(cstate) > nbytes)
+ /*
+ * If we now have at least sizeof(Vector8) input bytes ready, return
+ * them. This is beneficial for SIMD processing in the
+ * CopyReadLineText() function.
+ */
+ if (INPUT_BUF_BYTES(cstate) > nbytes + sizeof(Vector8))
return;
/*
@@ -1322,7 +1326,7 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
* unsafe with the old v2 COPY protocol, but we don't support that
* anymore.
*/
- if (input_buf_ptr >= copy_buf_len || need_data)
+ if (input_buf_ptr + sizeof(Vector8) >= copy_buf_len || need_data)
{
REFILL_LINEBUF;
@@ -1345,21 +1349,22 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
}
#ifndef USE_NO_SIMD
+
/*
- * SIMD instructions are used here to efficiently scan the input buffer
- * for special characters (e.g., newline, carriage return, quotes, or
- * escape characters). This approach significantly improves performance
- * compared to byte-by-byte iteration, especially for large input
- * buffers.
+ * SIMD instructions are used here to efficiently scan the input
+ * buffer for special characters (e.g., newline, carriage return,
+ * quotes, or escape characters). This approach significantly improves
+ * performance compared to byte-by-byte iteration, especially for
+ * large input buffers.
*
- * However, SIMD optimization cannot be applied in the following cases:
- * - Inside quoted fields, where escape sequences and closing quotes
- * require sequential processing to handle correctly.
- * - When the remaining buffer size is smaller than the size of a SIMD
- * vector register, as SIMD operations require processing data in
- * fixed-size chunks.
+ * However, SIMD optimization cannot be applied in the following
+ * cases: - Inside quoted fields, where escape sequences and closing
+ * quotes require sequential processing to handle correctly. - When
+ * the remaining buffer size is smaller than the size of a SIMD vector
+ * register, as SIMD operations require processing data in fixed-size
+ * chunks.
*/
- if (!in_quote && copy_buf_len - input_buf_ptr >= sizeof(Vector8))
+ if (copy_buf_len - input_buf_ptr >= sizeof(Vector8))
{
Vector8 chunk;
Vector8 match;
@@ -1388,13 +1393,15 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
* Found a special character. Advance up to that point and let
* the scalar code handle it.
*/
- int advance = pg_rightmost_one_pos32(mask);
+ int advance = pg_rightmost_one_pos32(mask);
+
input_buf_ptr += advance;
}
else
{
/* No special characters found, so skip the entire chunk */
input_buf_ptr += sizeof(Vector8);
+ last_was_esc = false;
continue;
}
}
@@ -1650,6 +1657,11 @@ CopyReadAttributesText(CopyFromState cstate)
char *cur_ptr;
char *line_end_ptr;
+#ifndef USE_NO_SIMD
+ Vector8 bs = vector8_broadcast('\\');
+ Vector8 delim = vector8_broadcast(delimc);
+#endif
+
/*
* We need a special case for zero-column tables: check that the input
* line is empty, and return.
@@ -1717,6 +1729,44 @@ CopyReadAttributesText(CopyFromState cstate)
{
char c;
+#ifndef USE_NO_SIMD
+ if (line_end_ptr - cur_ptr >= sizeof(Vector8))
+ {
+ Vector8 chunk;
+ Vector8 match;
+ uint32 mask;
+
+ /* Load a chunk of data into a vector register */
+ vector8_load(&chunk, (const uint8 *) cur_ptr);
+
+ /* Create a mask of all special characters we need to stop at */
+ match = vector8_or(vector8_eq(chunk, bs), vector8_eq(chunk, delim));
+
+ /* Check if we found any special characters */
+ mask = vector8_highbit_mask(match);
+ if (mask != 0)
+ {
+ /*
+ * Found a special character. Advance up to that point and
+ * let the scalar code handle it.
+ */
+ int advance = pg_rightmost_one_pos32(mask);
+
+ memcpy(output_ptr, cur_ptr, advance);
+ output_ptr += advance;
+ cur_ptr += advance;
+ }
+ else
+ {
+ /* No special characters found, so skip the entire chunk */
+ memcpy(output_ptr, cur_ptr, sizeof(Vector8));
+ output_ptr += sizeof(Vector8);
+ cur_ptr += sizeof(Vector8);
+ continue;
+ }
+ }
+#endif
+
end_ptr = cur_ptr;
if (cur_ptr >= line_end_ptr)
break;
@@ -1906,6 +1956,12 @@ CopyReadAttributesCSV(CopyFromState cstate)
char *cur_ptr;
char *line_end_ptr;
+#ifndef USE_NO_SIMD
+ Vector8 quote = vector8_broadcast(quotec);
+ Vector8 delim = vector8_broadcast(delimc);
+ Vector8 escape = vector8_broadcast(escapec);
+#endif
+
/*
* We need a special case for zero-column tables: check that the input
* line is empty, and return.
@@ -1972,6 +2028,50 @@ CopyReadAttributesCSV(CopyFromState cstate)
/* Not in quote */
for (;;)
{
+#ifndef USE_NO_SIMD
+ if (line_end_ptr - cur_ptr >= sizeof(Vector8))
+ {
+ Vector8 chunk;
+ Vector8 match;
+ uint32 mask;
+
+ /* Load a chunk of data into a vector register */
+ vector8_load(&chunk, (const uint8 *) cur_ptr);
+
+ /*
+ * Create a mask of all special characters we need to stop
+ * at
+ */
+ match = vector8_or(vector8_eq(chunk, quote), vector8_eq(chunk, delim));
+
+ /* Check if we found any special characters */
+ mask = vector8_highbit_mask(match);
+ if (mask != 0)
+ {
+ /*
+ * Found a special character. Advance up to that point
+ * and let the scalar code handle it.
+ */
+ int advance = pg_rightmost_one_pos32(mask);
+
+ memcpy(output_ptr, cur_ptr, advance);
+ output_ptr += advance;
+ cur_ptr += advance;
+ }
+ else
+ {
+ /*
+ * No special characters found, so skip the entire
+ * chunk
+ */
+ memcpy(output_ptr, cur_ptr, sizeof(Vector8));
+ output_ptr += sizeof(Vector8);
+ cur_ptr += sizeof(Vector8);
+ continue;
+ }
+ }
+#endif
+
end_ptr = cur_ptr;
if (cur_ptr >= line_end_ptr)
goto endfield;
@@ -1995,6 +2095,50 @@ CopyReadAttributesCSV(CopyFromState cstate)
/* In quote */
for (;;)
{
+#ifndef USE_NO_SIMD
+ if (line_end_ptr - cur_ptr >= sizeof(Vector8))
+ {
+ Vector8 chunk;
+ Vector8 match;
+ uint32 mask;
+
+ /* Load a chunk of data into a vector register */
+ vector8_load(&chunk, (const uint8 *) cur_ptr);
+
+ /*
+ * Create a mask of all special characters we need to stop
+ * at
+ */
+ match = vector8_or(vector8_eq(chunk, quote), vector8_eq(chunk, escape));
+
+ /* Check if we found any special characters */
+ mask = vector8_highbit_mask(match);
+ if (mask != 0)
+ {
+ /*
+ * Found a special character. Advance up to that point
+ * and let the scalar code handle it.
+ */
+ int advance = pg_rightmost_one_pos32(mask);
+
+ memcpy(output_ptr, cur_ptr, advance);
+ output_ptr += advance;
+ cur_ptr += advance;
+ }
+ else
+ {
+ /*
+ * No special characters found, so skip the entire
+ * chunk
+ */
+ memcpy(output_ptr, cur_ptr, sizeof(Vector8));
+ output_ptr += sizeof(Vector8);
+ cur_ptr += sizeof(Vector8);
+ continue;
+ }
+ }
+#endif
+
end_ptr = cur_ptr;
if (cur_ptr >= line_end_ptr)
ereport(ERROR,
--
2.50.1
Attachments:
[text/plain] v2-0001-Feedback.txt (7.9K, ../../CAN55FZ2AxiwSah7TiQoMB==r=JKT0bOtooCB7ov4xRrGkVmJ1A@mail.gmail.com/2-v2-0001-Feedback.txt)
download | inline diff:
From 203d648c4cf64c6d629f2abc719a371dd0393e22 Mon Sep 17 00:00:00 2001
From: Nazir Bilal Yavuz <byavuz81@gmail.com>
Date: Thu, 7 Aug 2025 13:27:34 +0300
Subject: [PATCH v2] Feedback
---
src/backend/commands/copyfromparse.c | 176 ++++++++++++++++++++++++---
1 file changed, 160 insertions(+), 16 deletions(-)
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index 5aba0fa6cb7..7b83e64e23b 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -670,8 +670,12 @@ CopyLoadInputBuf(CopyFromState cstate)
/* If we now have some unconverted data, try to convert it */
CopyConvertBuf(cstate);
- /* If we now have some more input bytes ready, return them */
- if (INPUT_BUF_BYTES(cstate) > nbytes)
+ /*
+ * If we now have at least sizeof(Vector8) input bytes ready, return
+ * them. This is beneficial for SIMD processing in the
+ * CopyReadLineText() function.
+ */
+ if (INPUT_BUF_BYTES(cstate) > nbytes + sizeof(Vector8))
return;
/*
@@ -1322,7 +1326,7 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
* unsafe with the old v2 COPY protocol, but we don't support that
* anymore.
*/
- if (input_buf_ptr >= copy_buf_len || need_data)
+ if (input_buf_ptr + sizeof(Vector8) >= copy_buf_len || need_data)
{
REFILL_LINEBUF;
@@ -1345,21 +1349,22 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
}
#ifndef USE_NO_SIMD
+
/*
- * SIMD instructions are used here to efficiently scan the input buffer
- * for special characters (e.g., newline, carriage return, quotes, or
- * escape characters). This approach significantly improves performance
- * compared to byte-by-byte iteration, especially for large input
- * buffers.
+ * SIMD instructions are used here to efficiently scan the input
+ * buffer for special characters (e.g., newline, carriage return,
+ * quotes, or escape characters). This approach significantly improves
+ * performance compared to byte-by-byte iteration, especially for
+ * large input buffers.
*
- * However, SIMD optimization cannot be applied in the following cases:
- * - Inside quoted fields, where escape sequences and closing quotes
- * require sequential processing to handle correctly.
- * - When the remaining buffer size is smaller than the size of a SIMD
- * vector register, as SIMD operations require processing data in
- * fixed-size chunks.
+ * However, SIMD optimization cannot be applied in the following
+ * cases: - Inside quoted fields, where escape sequences and closing
+ * quotes require sequential processing to handle correctly. - When
+ * the remaining buffer size is smaller than the size of a SIMD vector
+ * register, as SIMD operations require processing data in fixed-size
+ * chunks.
*/
- if (!in_quote && copy_buf_len - input_buf_ptr >= sizeof(Vector8))
+ if (copy_buf_len - input_buf_ptr >= sizeof(Vector8))
{
Vector8 chunk;
Vector8 match;
@@ -1388,13 +1393,15 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
* Found a special character. Advance up to that point and let
* the scalar code handle it.
*/
- int advance = pg_rightmost_one_pos32(mask);
+ int advance = pg_rightmost_one_pos32(mask);
+
input_buf_ptr += advance;
}
else
{
/* No special characters found, so skip the entire chunk */
input_buf_ptr += sizeof(Vector8);
+ last_was_esc = false;
continue;
}
}
@@ -1650,6 +1657,11 @@ CopyReadAttributesText(CopyFromState cstate)
char *cur_ptr;
char *line_end_ptr;
+#ifndef USE_NO_SIMD
+ Vector8 bs = vector8_broadcast('\\');
+ Vector8 delim = vector8_broadcast(delimc);
+#endif
+
/*
* We need a special case for zero-column tables: check that the input
* line is empty, and return.
@@ -1717,6 +1729,44 @@ CopyReadAttributesText(CopyFromState cstate)
{
char c;
+#ifndef USE_NO_SIMD
+ if (line_end_ptr - cur_ptr >= sizeof(Vector8))
+ {
+ Vector8 chunk;
+ Vector8 match;
+ uint32 mask;
+
+ /* Load a chunk of data into a vector register */
+ vector8_load(&chunk, (const uint8 *) cur_ptr);
+
+ /* Create a mask of all special characters we need to stop at */
+ match = vector8_or(vector8_eq(chunk, bs), vector8_eq(chunk, delim));
+
+ /* Check if we found any special characters */
+ mask = vector8_highbit_mask(match);
+ if (mask != 0)
+ {
+ /*
+ * Found a special character. Advance up to that point and
+ * let the scalar code handle it.
+ */
+ int advance = pg_rightmost_one_pos32(mask);
+
+ memcpy(output_ptr, cur_ptr, advance);
+ output_ptr += advance;
+ cur_ptr += advance;
+ }
+ else
+ {
+ /* No special characters found, so skip the entire chunk */
+ memcpy(output_ptr, cur_ptr, sizeof(Vector8));
+ output_ptr += sizeof(Vector8);
+ cur_ptr += sizeof(Vector8);
+ continue;
+ }
+ }
+#endif
+
end_ptr = cur_ptr;
if (cur_ptr >= line_end_ptr)
break;
@@ -1906,6 +1956,12 @@ CopyReadAttributesCSV(CopyFromState cstate)
char *cur_ptr;
char *line_end_ptr;
+#ifndef USE_NO_SIMD
+ Vector8 quote = vector8_broadcast(quotec);
+ Vector8 delim = vector8_broadcast(delimc);
+ Vector8 escape = vector8_broadcast(escapec);
+#endif
+
/*
* We need a special case for zero-column tables: check that the input
* line is empty, and return.
@@ -1972,6 +2028,50 @@ CopyReadAttributesCSV(CopyFromState cstate)
/* Not in quote */
for (;;)
{
+#ifndef USE_NO_SIMD
+ if (line_end_ptr - cur_ptr >= sizeof(Vector8))
+ {
+ Vector8 chunk;
+ Vector8 match;
+ uint32 mask;
+
+ /* Load a chunk of data into a vector register */
+ vector8_load(&chunk, (const uint8 *) cur_ptr);
+
+ /*
+ * Create a mask of all special characters we need to stop
+ * at
+ */
+ match = vector8_or(vector8_eq(chunk, quote), vector8_eq(chunk, delim));
+
+ /* Check if we found any special characters */
+ mask = vector8_highbit_mask(match);
+ if (mask != 0)
+ {
+ /*
+ * Found a special character. Advance up to that point
+ * and let the scalar code handle it.
+ */
+ int advance = pg_rightmost_one_pos32(mask);
+
+ memcpy(output_ptr, cur_ptr, advance);
+ output_ptr += advance;
+ cur_ptr += advance;
+ }
+ else
+ {
+ /*
+ * No special characters found, so skip the entire
+ * chunk
+ */
+ memcpy(output_ptr, cur_ptr, sizeof(Vector8));
+ output_ptr += sizeof(Vector8);
+ cur_ptr += sizeof(Vector8);
+ continue;
+ }
+ }
+#endif
+
end_ptr = cur_ptr;
if (cur_ptr >= line_end_ptr)
goto endfield;
@@ -1995,6 +2095,50 @@ CopyReadAttributesCSV(CopyFromState cstate)
/* In quote */
for (;;)
{
+#ifndef USE_NO_SIMD
+ if (line_end_ptr - cur_ptr >= sizeof(Vector8))
+ {
+ Vector8 chunk;
+ Vector8 match;
+ uint32 mask;
+
+ /* Load a chunk of data into a vector register */
+ vector8_load(&chunk, (const uint8 *) cur_ptr);
+
+ /*
+ * Create a mask of all special characters we need to stop
+ * at
+ */
+ match = vector8_or(vector8_eq(chunk, quote), vector8_eq(chunk, escape));
+
+ /* Check if we found any special characters */
+ mask = vector8_highbit_mask(match);
+ if (mask != 0)
+ {
+ /*
+ * Found a special character. Advance up to that point
+ * and let the scalar code handle it.
+ */
+ int advance = pg_rightmost_one_pos32(mask);
+
+ memcpy(output_ptr, cur_ptr, advance);
+ output_ptr += advance;
+ cur_ptr += advance;
+ }
+ else
+ {
+ /*
+ * No special characters found, so skip the entire
+ * chunk
+ */
+ memcpy(output_ptr, cur_ptr, sizeof(Vector8));
+ output_ptr += sizeof(Vector8);
+ cur_ptr += sizeof(Vector8);
+ continue;
+ }
+ }
+#endif
+
end_ptr = cur_ptr;
if (cur_ptr >= line_end_ptr)
ereport(ERROR,
--
2.50.1
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
@ 2025-08-12 07:25 ` Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
0 siblings, 1 reply; 178+ messages in thread
From: Shinya Kato @ 2025-08-12 07:25 UTC (permalink / raw)
To: Nazir Bilal Yavuz <byavuz81@gmail.com>; +Cc: pgsql-hackers
On Thu, Aug 7, 2025 at 8:15 PM Nazir Bilal Yavuz <byavuz81@gmail.com> wrote:
>
> Hi,
>
> Thank you for working on this!
>
> On Thu, 7 Aug 2025 at 04:49, Shinya Kato <shinya11.kato@gmail.com> wrote:
> >
> > Hi hackers,
> >
> > I have implemented SIMD optimization for the COPY FROM (FORMAT {csv,
> > text}) command and observed approximately a 5% performance
> > improvement. Please see the detailed test results below.
>
> I have been working on the same idea. I was not moving input_buf_ptr
> as far as possible, so I think your approach is better.
Great. I'm looking forward to working with you on this feature implementation.
> Also, I did a benchmark on text format. I created a benchmark for line
> length in a table being from 1 byte to 1 megabyte.The peak improvement
> is line length being 4096 and the improvement is more than 20% [1], I
> saw no regression on your patch.
Thank you for the additional benchmarks.
> I have a couple of ideas that I was working on:
> ---
>
> + * However, SIMD optimization cannot be applied in the following cases:
> + * - Inside quoted fields, where escape sequences and closing quotes
> + * require sequential processing to handle correctly.
>
> I think you can continue SIMD inside quoted fields. Only important
> thing is you need to set last_was_esc to false when SIMD skipped the
> chunk.
That's a clever point that last_was_esc should be reset to false when
a SIMD chunk is skipped. You're right about that specific case.
However, the core challenge is not what happens when we skip a chunk,
but what happens when a chunk contains special characters like quotes
or escapes. The main reason we avoid SIMD inside quoted fields is that
the parsing logic becomes fundamentally sequential and
context-dependent.
To correctly parse a "" as a single literal quote, we must perform a
lookahead to check the next character. This is an inherently
sequential operation that doesn't map well to SIMD's parallel nature.
Trying to handle this stateful logic with SIMD would lead to
significant implementation complexity, especially with edge cases like
an escape character falling on the last byte of a chunk.
> + * - When the remaining buffer size is smaller than the size of a SIMD
> + * vector register, as SIMD operations require processing data in
> + * fixed-size chunks.
>
> You run SIMD when 'copy_buf_len - input_buf_ptr >= sizeof(Vector8)'
> but you only call CopyLoadInputBuf() when 'input_buf_ptr >=
> copy_buf_len || need_data' so basically you need to wait at least the
> sizeof(Vector8) character to pass for the next SIMD. And in the worst
> case; if CopyLoadInputBuf() puts one character less than
> sizeof(Vector8), then you can't ever run SIMD. I think we need to make
> sure that CopyLoadInputBuf() loads at least the sizeof(Vector8)
> character to the input_buf so we do not encounter that problem.
I think you're probably right, but we only need to account for
sizeof(Vector8) when USE_NO_SIMD is not defined.
> What do you think about adding SIMD to CopyReadAttributesText() and
> CopyReadAttributesCSV() functions? When I add your SIMD approach to
> CopyReadAttributesText() function, the improvement on the 4096 byte
> line length input [1] goes from 20% to 30%.
Agreed, I will.
> I shared my ideas as a Feedback.txt file (.txt to stay off CFBot's
> radar for this thread). I hope these help, please let me know if you
> have any questions.
Thanks a lot!
On Mon, Aug 11, 2025 at 5:52 PM Nazir Bilal Yavuz <byavuz81@gmail.com> wrote:
>
> Hi,
>
> On Thu, 7 Aug 2025 at 14:15, Nazir Bilal Yavuz <byavuz81@gmail.com> wrote:
> >
> > On Thu, 7 Aug 2025 at 04:49, Shinya Kato <shinya11.kato@gmail.com> wrote:
> > >
> > > I have implemented SIMD optimization for the COPY FROM (FORMAT {csv,
> > > text}) command and observed approximately a 5% performance
> > > improvement. Please see the detailed test results below.
> >
> > Also, I did a benchmark on text format. I created a benchmark for line
> > length in a table being from 1 byte to 1 megabyte.The peak improvement
> > is line length being 4096 and the improvement is more than 20% [1], I
> > saw no regression on your patch.
>
> I did the same benchmark for the CSV format. The peak improvement is
> line length being 4096 and the improvement is more than 25% [1]. I saw
> a 5% regression on the 1 byte benchmark, there are no other
> regressions.
Thank you. I'm not too concerned about a regression when there's only
one byte per line.
> > What do you think about adding SIMD to CopyReadAttributesText() and
> > CopyReadAttributesCSV() functions? When I add your SIMD approach to
> > CopyReadAttributesText() function, the improvement on the 4096 byte
> > line length input [1] goes from 20% to 30%.
>
> I wanted to try using SIMD in CopyReadAttributesCSV() as well. The
> improvement on the 4096 byte line length input [1] goes from 25% to
> 35%, the regression on the 1 byte input is the same.
Yes, I'm on it. I'm currently adding the SIMD logic to
CopyReadAttributesCSV() as you suggested. I'll share the new version
of the patch soon.
--
Best regards,
Shinya Kato
NTT OSS Center
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
@ 2025-08-13 06:21 ` Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
0 siblings, 1 reply; 178+ messages in thread
From: Shinya Kato @ 2025-08-13 06:21 UTC (permalink / raw)
To: Nazir Bilal Yavuz <byavuz81@gmail.com>; +Cc: pgsql-hackers
On Tue, Aug 12, 2025 at 4:25 PM Shinya Kato <shinya11.kato@gmail.com> wrote:
> > + * However, SIMD optimization cannot be applied in the following cases:
> > + * - Inside quoted fields, where escape sequences and closing quotes
> > + * require sequential processing to handle correctly.
> >
> > I think you can continue SIMD inside quoted fields. Only important
> > thing is you need to set last_was_esc to false when SIMD skipped the
> > chunk.
>
> That's a clever point that last_was_esc should be reset to false when
> a SIMD chunk is skipped. You're right about that specific case.
>
> However, the core challenge is not what happens when we skip a chunk,
> but what happens when a chunk contains special characters like quotes
> or escapes. The main reason we avoid SIMD inside quoted fields is that
> the parsing logic becomes fundamentally sequential and
> context-dependent.
>
> To correctly parse a "" as a single literal quote, we must perform a
> lookahead to check the next character. This is an inherently
> sequential operation that doesn't map well to SIMD's parallel nature.
>
> Trying to handle this stateful logic with SIMD would lead to
> significant implementation complexity, especially with edge cases like
> an escape character falling on the last byte of a chunk.
Ah, you're right. My apologies, I misunderstood the implementation. It
appears that SIMD can be used even within quoted strings.
I think it would be better not to use the SIMD path when last_was_esc
is true. The next character is likely to be a special character, and
handling this case outside the SIMD loop would also improve
readability by consolidating the last_was_esc toggle logic in one
place.
Furthermore, when inside a quote (in_quote) in CSV mode, the detection
of \n and \r can be disabled.
+ last_was_esc = false;
Regarding the implementation, I believe we must set last_was_esc to
false when advancing input_buf_ptr, as shown in the code below. For
this reason, I think it’s best to keep the current logic for toggling
last_was_esc.
+ int advance = pg_rightmost_one_pos32(mask);
+ input_buf_ptr += advance;
I've attached a new patch that includes these changes. Further
modifications are still in progress.
--
Best regards,
Shinya Kato
NTT OSS Center
Attachments:
[application/octet-stream] v2-0001-Speed-up-COPY-FROM-text-CSV-parsing-using-SIMD.patch (3.4K, ../../CAOzEurSqgA69er9SzhPnXwmsVpO7-piUOuOy3dXcHOi__nSQcg@mail.gmail.com/2-v2-0001-Speed-up-COPY-FROM-text-CSV-parsing-using-SIMD.patch)
download | inline diff:
From 69e16f8c7a52d967385a1dc9b1602bbd4472df60 Mon Sep 17 00:00:00 2001
From: Shinya Kato <shinya11.kato@gmail.com>
Date: Mon, 28 Jul 2025 22:08:20 +0900
Subject: [PATCH v2] Speed up COPY FROM text/CSV parsing using SIMD
---
src/backend/commands/copyfromparse.c | 71 ++++++++++++++++++++++++++++
1 file changed, 71 insertions(+)
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index b1ae97b833d..f1a6ea81dd1 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -71,7 +71,9 @@
#include "mb/pg_wchar.h"
#include "miscadmin.h"
#include "pgstat.h"
+#include "port/pg_bitutils.h"
#include "port/pg_bswap.h"
+#include "port/simd.h"
#include "utils/builtins.h"
#include "utils/rel.h"
@@ -1255,6 +1257,14 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
char quotec = '\0';
char escapec = '\0';
+#ifndef USE_NO_SIMD
+ Vector8 nl = vector8_broadcast('\n');
+ Vector8 cr = vector8_broadcast('\r');
+ Vector8 bs = vector8_broadcast('\\');
+ Vector8 quote;
+ Vector8 escape;
+#endif
+
if (is_csv)
{
quotec = cstate->opts.quote[0];
@@ -1262,6 +1272,12 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
/* ignore special escape processing if it's the same as quotec */
if (quotec == escapec)
escapec = '\0';
+
+#ifndef USE_NO_SIMD
+ quote = vector8_broadcast(quotec);
+ if (quotec != escapec)
+ escape = vector8_broadcast(escapec);
+#endif
}
/*
@@ -1328,6 +1344,61 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
need_data = false;
}
+#ifndef USE_NO_SIMD
+ /*
+ * Use SIMD instructions to efficiently scan the input buffer for
+ * special characters (e.g., newline, carriage return, quote, and
+ * escape). This is faster than byte-by-byte iteration, especially on
+ * large buffers.
+ *
+ * We do not apply the SIMD fast path in either of the following cases:
+ * - When the previously processed character was an escape character
+ * (last_was_esc), since the next byte must be examined sequentially.
+ * - The remaining buffer is smaller than one vector width
+ * (sizeof(Vector8)); SIMD operates on fixed-size chunks.
+ */
+ if (!last_was_esc && copy_buf_len - input_buf_ptr >= sizeof(Vector8))
+ {
+ Vector8 chunk;
+ Vector8 match;
+ uint32 mask;
+
+ /* Load a chunk of data into a vector register */
+ vector8_load(&chunk, (const uint8 *) ©_input_buf[input_buf_ptr]);
+
+ /* \n and \r are not special inside quotes */
+ if (!in_quote)
+ match = vector8_or(vector8_eq(chunk, nl), vector8_eq(chunk, cr));
+
+ if (is_csv)
+ {
+ match = vector8_or(match, vector8_eq(chunk, quote));
+ if (escapec != '\0')
+ match = vector8_or(match, vector8_eq(chunk, escape));
+ }
+ else
+ match = vector8_or(match, vector8_eq(chunk, bs));
+
+ /* Check if we found any special characters */
+ mask = vector8_highbit_mask(match);
+ if (mask != 0)
+ {
+ /*
+ * Found a special character. Advance up to that point and let
+ * the scalar code handle it.
+ */
+ int advance = pg_rightmost_one_pos32(mask);
+ input_buf_ptr += advance;
+ }
+ else
+ {
+ /* No special characters found, so skip the entire chunk */
+ input_buf_ptr += sizeof(Vector8);
+ continue;
+ }
+ }
+#endif
+
/* OK to fetch a character */
prev_raw_ptr = input_buf_ptr;
c = copy_input_buf[input_buf_ptr++];
--
2.47.3
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
@ 2025-08-14 02:24 ` KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
0 siblings, 1 reply; 178+ messages in thread
From: KAZAR Ayoub @ 2025-08-14 02:24 UTC (permalink / raw)
To: Shinya Kato <shinya11.kato@gmail.com>; +Cc: Nazir Bilal Yavuz <byavuz81@gmail.com>; pgsql-hackers
Following Nazir's findings about 4096 bytes being the performant line
length, I did more benchmarks from my side on both TEXT and CSV formats
with two different cases of normal data (no special characters) and data
with many special characters.
Results are con good as expected and similar to previous benchmarks
~30.9% faster copy in TEXT format
~32.4% faster copy in CSV format
20%-30% reduces cycles per instructions
In the case of doing a lot of special characters in the lines (e.g., tables
with large numbers of columns maybe), we obviously expect regressions here
because of the overhead of many fallbacks to scalar processing.
Results for a 1/3 of line length of special characters:
~43.9% slower copy in TEXT format
~16.7% slower copy in CSV format
So for even less occurrences of special characters or wider distance
between there might still be some regressions in this case, a
non-significant case maybe, but can be treated in other patches if we
consider to not use SIMD path sometimes.
I hope this helps more and confirms the patch.
Regards,
Ayoub Kazar
Le jeu. 14 août 2025 à 01:55, Shinya Kato <shinya11.kato@gmail.com> a
écrit :
> On Tue, Aug 12, 2025 at 4:25 PM Shinya Kato <shinya11.kato@gmail.com>
> wrote:
>
> > > + * However, SIMD optimization cannot be applied in the
> following cases:
> > > + * - Inside quoted fields, where escape sequences and closing
> quotes
> > > + * require sequential processing to handle correctly.
> > >
> > > I think you can continue SIMD inside quoted fields. Only important
> > > thing is you need to set last_was_esc to false when SIMD skipped the
> > > chunk.
> >
> > That's a clever point that last_was_esc should be reset to false when
> > a SIMD chunk is skipped. You're right about that specific case.
> >
> > However, the core challenge is not what happens when we skip a chunk,
> > but what happens when a chunk contains special characters like quotes
> > or escapes. The main reason we avoid SIMD inside quoted fields is that
> > the parsing logic becomes fundamentally sequential and
> > context-dependent.
> >
> > To correctly parse a "" as a single literal quote, we must perform a
> > lookahead to check the next character. This is an inherently
> > sequential operation that doesn't map well to SIMD's parallel nature.
> >
> > Trying to handle this stateful logic with SIMD would lead to
> > significant implementation complexity, especially with edge cases like
> > an escape character falling on the last byte of a chunk.
>
> Ah, you're right. My apologies, I misunderstood the implementation. It
> appears that SIMD can be used even within quoted strings.
>
> I think it would be better not to use the SIMD path when last_was_esc
> is true. The next character is likely to be a special character, and
> handling this case outside the SIMD loop would also improve
> readability by consolidating the last_was_esc toggle logic in one
> place.
>
> Furthermore, when inside a quote (in_quote) in CSV mode, the detection
> of \n and \r can be disabled.
>
> + last_was_esc = false;
>
> Regarding the implementation, I believe we must set last_was_esc to
> false when advancing input_buf_ptr, as shown in the code below. For
> this reason, I think it’s best to keep the current logic for toggling
> last_was_esc.
>
> + int advance = pg_rightmost_one_pos32(mask);
> + input_buf_ptr += advance;
>
> I've attached a new patch that includes these changes. Further
> modifications are still in progress.
>
> --
> Best regards,
> Shinya Kato
> NTT OSS Center
>
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
@ 2025-08-14 10:29 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
0 siblings, 1 reply; 178+ messages in thread
From: Nazir Bilal Yavuz @ 2025-08-14 10:29 UTC (permalink / raw)
To: KAZAR Ayoub <ma_kazar@esi.dz>; +Cc: Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
Hi,
On Thu, 14 Aug 2025 at 05:25, KAZAR Ayoub <ma_kazar@esi.dz> wrote:
>
> Following Nazir's findings about 4096 bytes being the performant line length, I did more benchmarks from my side on both TEXT and CSV formats with two different cases of normal data (no special characters) and data with many special characters.
>
> Results are con good as expected and similar to previous benchmarks
> ~30.9% faster copy in TEXT format
> ~32.4% faster copy in CSV format
> 20%-30% reduces cycles per instructions
>
> In the case of doing a lot of special characters in the lines (e.g., tables with large numbers of columns maybe), we obviously expect regressions here because of the overhead of many fallbacks to scalar processing.
> Results for a 1/3 of line length of special characters:
> ~43.9% slower copy in TEXT format
> ~16.7% slower copy in CSV format
> So for even less occurrences of special characters or wider distance between there might still be some regressions in this case, a non-significant case maybe, but can be treated in other patches if we consider to not use SIMD path sometimes.
>
> I hope this helps more and confirms the patch.
Thanks for running that benchmark! Would you mind sharing a reproducer
for the regression you observed?
--
Regards,
Nazir Bilal Yavuz
Microsoft
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
@ 2025-08-14 14:59 ` KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
0 siblings, 1 reply; 178+ messages in thread
From: KAZAR Ayoub @ 2025-08-14 14:59 UTC (permalink / raw)
To: Nazir Bilal Yavuz <byavuz81@gmail.com>; +Cc: Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
> Hi,
>
> On Thu, 14 Aug 2025 at 05:25, KAZAR Ayoub <ma_kazar@esi.dz> wrote:
> >
> > Following Nazir's findings about 4096 bytes being the performant line
> length, I did more benchmarks from my side on both TEXT and CSV formats
> with two different cases of normal data (no special characters) and data
> with many special characters.
> >
> > Results are con good as expected and similar to previous benchmarks
> > ~30.9% faster copy in TEXT format
> > ~32.4% faster copy in CSV format
> > 20%-30% reduces cycles per instructions
> >
> > In the case of doing a lot of special characters in the lines (e.g.,
> tables with large numbers of columns maybe), we obviously expect
> regressions here because of the overhead of many fallbacks to scalar
> processing.
> > Results for a 1/3 of line length of special characters:
> > ~43.9% slower copy in TEXT format
> > ~16.7% slower copy in CSV format
> > So for even less occurrences of special characters or wider distance
> between there might still be some regressions in this case, a
> non-significant case maybe, but can be treated in other patches if we
> consider to not use SIMD path sometimes.
> >
> > I hope this helps more and confirms the patch.
>
> Thanks for running that benchmark! Would you mind sharing a reproducer
> for the regression you observed?
>
> --
> Regards,
> Nazir Bilal Yavuz
> Microsoft
Of course, I attached the sql to generate the text and csv test files.
If having a 1/3 of line length of special characters can be an
exaggeration, something lower might still reproduce some regressions of
course for the same idea.
Best regards,
Ayoub Kazar
Attachments:
[application/sql] simd-copy-from-bench.sql (812B, ../../CA+K2Ru=jHuz_Wpgar4Sobtxeb33qxx=o59ToOhZ=vpmkMqErnA@mail.gmail.com/3-simd-copy-from-bench.sql)
download
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
@ 2025-08-19 12:33 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 19:36 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
0 siblings, 2 replies; 178+ messages in thread
From: Nazir Bilal Yavuz @ 2025-08-19 12:33 UTC (permalink / raw)
To: KAZAR Ayoub <ma_kazar@esi.dz>; +Cc: Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
Hi,
On Thu, 14 Aug 2025 at 18:00, KAZAR Ayoub <ma_kazar@esi.dz> wrote:
>> Thanks for running that benchmark! Would you mind sharing a reproducer
>> for the regression you observed?
>
> Of course, I attached the sql to generate the text and csv test files.
> If having a 1/3 of line length of special characters can be an exaggeration, something lower might still reproduce some regressions of course for the same idea.
Thank you so much!
I am able to reproduce the regression you mentioned but both
regressions are %20 on my end. I found that (by experimenting) SIMD
causes a regression if it advances less than 5 characters.
So, I implemented a small heuristic. It works like that:
- If advance < 5 -> insert a sleep penalty (n cycles).
- Each time advance < 5, n is doubled.
- Each time advance ≥ 5, n is halved.
I am sharing a POC patch to show heuristic, it can be applied on top
of v1-0001. Heuristic version has the same performance improvements
with the v1-0001 but the regression is %5 instead of %20 compared to
the master.
--
Regards,
Nazir Bilal Yavuz
Microsoft
From aa55843b0c64bed9f72cf8cd7854df9df7ef989b Mon Sep 17 00:00:00 2001
From: Nazir Bilal Yavuz <byavuz81@gmail.com>
Date: Tue, 19 Aug 2025 15:16:02 +0300
Subject: [PATCH v1] COPY SIMD: add heuristic to avoid regression on small
advances
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
When SIMD advances fewer than 5 characters, performance regresses.
To mitigate this, introduce a heuristic:
- If advance < 5 -> insert a sleep penalty (n cycles).
- Each time advance < 5, n is doubled.
- Each time advance ≥ 5, n is halved.
---
src/backend/commands/copyfromparse.c | 42 ++++++++++++++++++++++++++--
1 file changed, 40 insertions(+), 2 deletions(-)
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index 5aba0fa6cb7..e58d7d4e353 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -1263,6 +1263,9 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
Vector8 bs = vector8_broadcast('\\');
Vector8 quote;
Vector8 escape;
+
+ int sleep_cyle = 0;
+ int last_sleep_cyle = 1;
#endif
if (is_csv)
@@ -1359,7 +1362,7 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
* vector register, as SIMD operations require processing data in
* fixed-size chunks.
*/
- if (!in_quote && copy_buf_len - input_buf_ptr >= sizeof(Vector8))
+ if (sleep_cyle <= 0 && !in_quote && copy_buf_len - input_buf_ptr >= sizeof(Vector8))
{
Vector8 chunk;
Vector8 match;
@@ -1390,14 +1393,49 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
*/
int advance = pg_rightmost_one_pos32(mask);
input_buf_ptr += advance;
+
+ /*
+ * If we advance less than 5 characters we cause regression.
+ * Sleep a bit then try again. Sleep time increases
+ * exponentially.
+ */
+ if (advance < 5)
+ {
+ if (last_sleep_cyle >= PG_INT16_MAX / 2)
+ last_sleep_cyle = PG_INT16_MAX;
+ else
+ last_sleep_cyle = last_sleep_cyle << 1;
+
+ sleep_cyle = last_sleep_cyle;
+ }
+
+ /*
+ * If we advance more than 4 charactes this means we have
+ * performance improvement. Halve sleep time for next sleep.
+ */
+ else
+ {
+ last_sleep_cyle = Max(last_sleep_cyle >> 1, 1);
+ sleep_cyle = 0;
+ }
}
else
{
- /* No special characters found, so skip the entire chunk */
+ /*
+ * No special characters found, so skip the entire chunk and
+ * halve sleep time for next sleep.
+ */
input_buf_ptr += sizeof(Vector8);
+ last_sleep_cyle = Max(last_sleep_cyle >> 1, 1);
continue;
}
}
+
+ /*
+ * Vulnerable to overflow if we are in quote for more than INT16_MAX
+ * characters.
+ */
+ sleep_cyle--;
#endif
/* OK to fetch a character */
--
2.50.1
Attachments:
[text/plain] COPY-SIMD-add-heuristic-to-avoid-regression-on-sm.txt (2.8K, ../../CAN55FZ1J+6eM=F5GreWEBMJcNV_gifYyYY1b6xpYzun=nWPhMQ@mail.gmail.com/2-COPY-SIMD-add-heuristic-to-avoid-regression-on-sm.txt)
download | inline diff:
From aa55843b0c64bed9f72cf8cd7854df9df7ef989b Mon Sep 17 00:00:00 2001
From: Nazir Bilal Yavuz <byavuz81@gmail.com>
Date: Tue, 19 Aug 2025 15:16:02 +0300
Subject: [PATCH v1] COPY SIMD: add heuristic to avoid regression on small
advances
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
When SIMD advances fewer than 5 characters, performance regresses.
To mitigate this, introduce a heuristic:
- If advance < 5 -> insert a sleep penalty (n cycles).
- Each time advance < 5, n is doubled.
- Each time advance ≥ 5, n is halved.
---
src/backend/commands/copyfromparse.c | 42 ++++++++++++++++++++++++++--
1 file changed, 40 insertions(+), 2 deletions(-)
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index 5aba0fa6cb7..e58d7d4e353 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -1263,6 +1263,9 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
Vector8 bs = vector8_broadcast('\\');
Vector8 quote;
Vector8 escape;
+
+ int sleep_cyle = 0;
+ int last_sleep_cyle = 1;
#endif
if (is_csv)
@@ -1359,7 +1362,7 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
* vector register, as SIMD operations require processing data in
* fixed-size chunks.
*/
- if (!in_quote && copy_buf_len - input_buf_ptr >= sizeof(Vector8))
+ if (sleep_cyle <= 0 && !in_quote && copy_buf_len - input_buf_ptr >= sizeof(Vector8))
{
Vector8 chunk;
Vector8 match;
@@ -1390,14 +1393,49 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
*/
int advance = pg_rightmost_one_pos32(mask);
input_buf_ptr += advance;
+
+ /*
+ * If we advance less than 5 characters we cause regression.
+ * Sleep a bit then try again. Sleep time increases
+ * exponentially.
+ */
+ if (advance < 5)
+ {
+ if (last_sleep_cyle >= PG_INT16_MAX / 2)
+ last_sleep_cyle = PG_INT16_MAX;
+ else
+ last_sleep_cyle = last_sleep_cyle << 1;
+
+ sleep_cyle = last_sleep_cyle;
+ }
+
+ /*
+ * If we advance more than 4 charactes this means we have
+ * performance improvement. Halve sleep time for next sleep.
+ */
+ else
+ {
+ last_sleep_cyle = Max(last_sleep_cyle >> 1, 1);
+ sleep_cyle = 0;
+ }
}
else
{
- /* No special characters found, so skip the entire chunk */
+ /*
+ * No special characters found, so skip the entire chunk and
+ * halve sleep time for next sleep.
+ */
input_buf_ptr += sizeof(Vector8);
+ last_sleep_cyle = Max(last_sleep_cyle >> 1, 1);
continue;
}
}
+
+ /*
+ * Vulnerable to overflow if we are in quote for more than INT16_MAX
+ * characters.
+ */
+ sleep_cyle--;
#endif
/* OK to fetch a character */
--
2.50.1
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
@ 2025-08-19 14:14 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
1 sibling, 1 reply; 178+ messages in thread
From: Nazir Bilal Yavuz @ 2025-08-19 14:14 UTC (permalink / raw)
To: KAZAR Ayoub <ma_kazar@esi.dz>; +Cc: Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
Hi,
On Tue, 19 Aug 2025 at 15:33, Nazir Bilal Yavuz <byavuz81@gmail.com> wrote:
>
> I am able to reproduce the regression you mentioned but both
> regressions are %20 on my end. I found that (by experimenting) SIMD
> causes a regression if it advances less than 5 characters.
>
> So, I implemented a small heuristic. It works like that:
>
> - If advance < 5 -> insert a sleep penalty (n cycles).
'sleep' might be a poor word choice here. I meant skipping SIMD for n
number of times.
--
Regards,
Nazir Bilal Yavuz
Microsoft
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
@ 2025-08-21 15:47 ` Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
0 siblings, 1 reply; 178+ messages in thread
From: Andrew Dunstan @ 2025-08-21 15:47 UTC (permalink / raw)
To: Nazir Bilal Yavuz <byavuz81@gmail.com>; KAZAR Ayoub <ma_kazar@esi.dz>; +Cc: Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
On 2025-08-19 Tu 10:14 AM, Nazir Bilal Yavuz wrote:
> Hi,
>
> On Tue, 19 Aug 2025 at 15:33, Nazir Bilal Yavuz <byavuz81@gmail.com> wrote:
>> I am able to reproduce the regression you mentioned but both
>> regressions are %20 on my end. I found that (by experimenting) SIMD
>> causes a regression if it advances less than 5 characters.
>>
>> So, I implemented a small heuristic. It works like that:
>>
>> - If advance < 5 -> insert a sleep penalty (n cycles).
> 'sleep' might be a poor word choice here. I meant skipping SIMD for n
> number of times.
>
I was thinking a bit about that this morning. I wonder if it might be
better instead of having a constantly applied heuristic like this, it
might be better to do a little extra accounting in the first, say, 1000
lines of an input file, and if less than some portion of the input is
found to be special characters then switch to the SIMD code. What that
portion should be would need to be determined by some experimentation
with a variety of typical workloads, but given your findings 20% seems
like a good starting point.
cheers
andrew
--
Andrew Dunstan
EDB: https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
@ 2025-10-16 14:29 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-18 18:46 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-10-18 20:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
0 siblings, 3 replies; 178+ messages in thread
From: Nazir Bilal Yavuz @ 2025-10-16 14:29 UTC (permalink / raw)
To: Andrew Dunstan <andrew@dunslane.net>; +Cc: KAZAR Ayoub <ma_kazar@esi.dz>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
Hi,
On Thu, 21 Aug 2025 at 18:47, Andrew Dunstan <andrew@dunslane.net> wrote:
>
>
> On 2025-08-19 Tu 10:14 AM, Nazir Bilal Yavuz wrote:
> > Hi,
> >
> > On Tue, 19 Aug 2025 at 15:33, Nazir Bilal Yavuz <byavuz81@gmail.com> wrote:
> >> I am able to reproduce the regression you mentioned but both
> >> regressions are %20 on my end. I found that (by experimenting) SIMD
> >> causes a regression if it advances less than 5 characters.
> >>
> >> So, I implemented a small heuristic. It works like that:
> >>
> >> - If advance < 5 -> insert a sleep penalty (n cycles).
> > 'sleep' might be a poor word choice here. I meant skipping SIMD for n
> > number of times.
> >
>
> I was thinking a bit about that this morning. I wonder if it might be
> better instead of having a constantly applied heuristic like this, it
> might be better to do a little extra accounting in the first, say, 1000
> lines of an input file, and if less than some portion of the input is
> found to be special characters then switch to the SIMD code. What that
> portion should be would need to be determined by some experimentation
> with a variety of typical workloads, but given your findings 20% seems
> like a good starting point.
I implemented a heuristic something similar to this. It is a mix of
previous heuristic and your idea, it works like that:
Overall logic is that we will not run SIMD for the entire line and we
decide if it is worth it to run SIMD for the next lines.
1 - We will try SIMD and decide if it is worth it to run SIMD.
1.1 - If it is worth it, we will continue to run SIMD and we will
halve the simd_last_sleep_cycle variable.
1.2 - If it is not worth it, we will double the simd_last_sleep_cycle
and we will not run SIMD for these many lines.
1.3 - After skipping simd_last_sleep_cycle lines, we will go back to the #1.
Note: simd_last_sleep_cycle can not pass 1024, so we will run SIMD for
each 1024 lines at max.
With this heuristic the regression is limited by %2 in the worst case.
Patches are attached, the first patch is v2-0001 from Shinya with the
'-Werror=maybe-uninitialized' fixes and the pgindent changes. 0002 is
the actual heuristic patch.
--
Regards,
Nazir Bilal Yavuz
Microsoft
Attachments:
[text/x-patch] v3-0001-Speed-up-COPY-FROM-text-CSV-parsing-using-SIMD.patch (3.4K, ../../CAN55FZ1KF7XNpm2XyG=M-sFUODai=6Z8a11xE3s4YRBeBKY3tA@mail.gmail.com/2-v3-0001-Speed-up-COPY-FROM-text-CSV-parsing-using-SIMD.patch)
download | inline diff:
From 2d2372e90305a81c80fe182003933039bf32f97e Mon Sep 17 00:00:00 2001
From: Shinya Kato <shinya11.kato@gmail.com>
Date: Mon, 28 Jul 2025 22:08:20 +0900
Subject: [PATCH v3 1/2] Speed up COPY FROM text/CSV parsing using SIMD
---
src/backend/commands/copyfromparse.c | 73 ++++++++++++++++++++++++++++
1 file changed, 73 insertions(+)
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index b1ae97b833d..99959a40fab 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -71,7 +71,9 @@
#include "mb/pg_wchar.h"
#include "miscadmin.h"
#include "pgstat.h"
+#include "port/pg_bitutils.h"
#include "port/pg_bswap.h"
+#include "port/simd.h"
#include "utils/builtins.h"
#include "utils/rel.h"
@@ -1255,6 +1257,14 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
char quotec = '\0';
char escapec = '\0';
+#ifndef USE_NO_SIMD
+ Vector8 nl = vector8_broadcast('\n');
+ Vector8 cr = vector8_broadcast('\r');
+ Vector8 bs = vector8_broadcast('\\');
+ Vector8 quote = vector8_broadcast(0);
+ Vector8 escape = vector8_broadcast(0);
+#endif
+
if (is_csv)
{
quotec = cstate->opts.quote[0];
@@ -1262,6 +1272,12 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
/* ignore special escape processing if it's the same as quotec */
if (quotec == escapec)
escapec = '\0';
+
+#ifndef USE_NO_SIMD
+ quote = vector8_broadcast(quotec);
+ if (quotec != escapec)
+ escape = vector8_broadcast(escapec);
+#endif
}
/*
@@ -1328,6 +1344,63 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
need_data = false;
}
+#ifndef USE_NO_SIMD
+
+ /*
+ * Use SIMD instructions to efficiently scan the input buffer for
+ * special characters (e.g., newline, carriage return, quote, and
+ * escape). This is faster than byte-by-byte iteration, especially on
+ * large buffers.
+ *
+ * We do not apply the SIMD fast path in either of the following
+ * cases: - When the previously processed character was an escape
+ * character (last_was_esc), since the next byte must be examined
+ * sequentially. - The remaining buffer is smaller than one vector
+ * width (sizeof(Vector8)); SIMD operates on fixed-size chunks.
+ */
+ if (!last_was_esc && copy_buf_len - input_buf_ptr >= sizeof(Vector8))
+ {
+ Vector8 chunk;
+ Vector8 match = vector8_broadcast(0);
+ uint32 mask;
+
+ /* Load a chunk of data into a vector register */
+ vector8_load(&chunk, (const uint8 *) ©_input_buf[input_buf_ptr]);
+
+ /* \n and \r are not special inside quotes */
+ if (!in_quote)
+ match = vector8_or(vector8_eq(chunk, nl), vector8_eq(chunk, cr));
+
+ if (is_csv)
+ {
+ match = vector8_or(match, vector8_eq(chunk, quote));
+ if (escapec != '\0')
+ match = vector8_or(match, vector8_eq(chunk, escape));
+ }
+ else
+ match = vector8_or(match, vector8_eq(chunk, bs));
+
+ /* Check if we found any special characters */
+ mask = vector8_highbit_mask(match);
+ if (mask != 0)
+ {
+ /*
+ * Found a special character. Advance up to that point and let
+ * the scalar code handle it.
+ */
+ int advance = pg_rightmost_one_pos32(mask);
+
+ input_buf_ptr += advance;
+ }
+ else
+ {
+ /* No special characters found, so skip the entire chunk */
+ input_buf_ptr += sizeof(Vector8);
+ continue;
+ }
+ }
+#endif
+
/* OK to fetch a character */
prev_raw_ptr = input_buf_ptr;
c = copy_input_buf[input_buf_ptr++];
--
2.51.0
[text/x-patch] v3-0002-COPY-SIMD-per-line-heuristic.patch (6.5K, ../../CAN55FZ1KF7XNpm2XyG=M-sFUODai=6Z8a11xE3s4YRBeBKY3tA@mail.gmail.com/3-v3-0002-COPY-SIMD-per-line-heuristic.patch)
download | inline diff:
From ad050583d3c14bdec44266d8d2110b384fa9d7dc Mon Sep 17 00:00:00 2001
From: Nazir Bilal Yavuz <byavuz81@gmail.com>
Date: Tue, 14 Oct 2025 13:18:13 +0300
Subject: [PATCH v3 2/2] COPY SIMD per-line heuristic
---
src/include/commands/copyfrom_internal.h | 7 ++
src/backend/commands/copyfrom.c | 6 ++
src/backend/commands/copyfromparse.c | 82 ++++++++++++++++++++++--
3 files changed, 89 insertions(+), 6 deletions(-)
diff --git a/src/include/commands/copyfrom_internal.h b/src/include/commands/copyfrom_internal.h
index c8b22af22d8..9dd31320f52 100644
--- a/src/include/commands/copyfrom_internal.h
+++ b/src/include/commands/copyfrom_internal.h
@@ -89,6 +89,13 @@ typedef struct CopyFromStateData
const char *cur_attval; /* current att value for error messages */
bool relname_only; /* don't output line number, att, etc. */
+ /* SIMD variables */
+ bool simd_continue;
+ bool simd_initialized;
+ uint16 simd_last_sleep_cycle;
+ uint16 simd_current_sleep_cycle;
+
+
/*
* Working state
*/
diff --git a/src/backend/commands/copyfrom.c b/src/backend/commands/copyfrom.c
index 12781963b4f..4bdfd96c244 100644
--- a/src/backend/commands/copyfrom.c
+++ b/src/backend/commands/copyfrom.c
@@ -1721,6 +1721,12 @@ BeginCopyFrom(ParseState *pstate,
cstate->cur_attval = NULL;
cstate->relname_only = false;
+ /* Initialize SIMD variables */
+ cstate->simd_continue = false;
+ cstate->simd_initialized = false;
+ cstate->simd_current_sleep_cycle = 0;
+ cstate->simd_last_sleep_cycle = 0;
+
/*
* Allocate buffers for the input pipeline.
*
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index 99959a40fab..24cef54e5e4 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -143,12 +143,14 @@ static const char BinarySignature[11] = "PGCOPY\n\377\r\n\0";
/* non-export function prototypes */
static bool CopyReadLine(CopyFromState cstate, bool is_csv);
-static bool CopyReadLineText(CopyFromState cstate, bool is_csv);
static int CopyReadAttributesText(CopyFromState cstate);
static int CopyReadAttributesCSV(CopyFromState cstate);
static Datum CopyReadBinaryAttribute(CopyFromState cstate, FmgrInfo *flinfo,
Oid typioparam, int32 typmod,
bool *isnull);
+static pg_attribute_always_inline bool CopyReadLineText(CopyFromState cstate,
+ bool is_csv,
+ bool simd_continue);
static pg_attribute_always_inline bool CopyFromTextLikeOneRow(CopyFromState cstate,
ExprContext *econtext,
Datum *values,
@@ -1173,8 +1175,23 @@ CopyReadLine(CopyFromState cstate, bool is_csv)
resetStringInfo(&cstate->line_buf);
cstate->line_buf_valid = false;
- /* Parse data and transfer into line_buf */
- result = CopyReadLineText(cstate, is_csv);
+ /* If that is the first time we do read, initalize the SIMD */
+ if (unlikely(!cstate->simd_initialized))
+ {
+ cstate->simd_initialized = true;
+ cstate->simd_continue = true;
+ cstate->simd_current_sleep_cycle = 0;
+ cstate->simd_last_sleep_cycle = 0;
+ }
+
+ /*
+ * Parse data and transfer into line_buf. To get benefit from inlining,
+ * call CopyReadLineText() with the constant boolean variables.
+ */
+ if (cstate->simd_continue)
+ result = CopyReadLineText(cstate, is_csv, true);
+ else
+ result = CopyReadLineText(cstate, is_csv, false);
if (result)
{
@@ -1241,8 +1258,8 @@ CopyReadLine(CopyFromState cstate, bool is_csv)
/*
* CopyReadLineText - inner loop of CopyReadLine for text mode
*/
-static bool
-CopyReadLineText(CopyFromState cstate, bool is_csv)
+static pg_attribute_always_inline bool
+CopyReadLineText(CopyFromState cstate, bool is_csv, bool simd_continue)
{
char *copy_input_buf;
int input_buf_ptr;
@@ -1258,11 +1275,16 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
char escapec = '\0';
#ifndef USE_NO_SIMD
+#define SIMD_SLEEP_MAX 1024
+#define SIMD_ADVANCE_AT_LEAST 5
Vector8 nl = vector8_broadcast('\n');
Vector8 cr = vector8_broadcast('\r');
Vector8 bs = vector8_broadcast('\\');
Vector8 quote = vector8_broadcast(0);
Vector8 escape = vector8_broadcast(0);
+
+ uint64 simd_total_cycle = 0;
+ uint64 simd_total_advance = 0;
#endif
if (is_csv)
@@ -1358,12 +1380,14 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
* sequentially. - The remaining buffer is smaller than one vector
* width (sizeof(Vector8)); SIMD operates on fixed-size chunks.
*/
- if (!last_was_esc && copy_buf_len - input_buf_ptr >= sizeof(Vector8))
+ if (simd_continue && !last_was_esc && copy_buf_len - input_buf_ptr >= sizeof(Vector8))
{
Vector8 chunk;
Vector8 match = vector8_broadcast(0);
uint32 mask;
+ simd_total_cycle++;
+
/* Load a chunk of data into a vector register */
vector8_load(&chunk, (const uint8 *) ©_input_buf[input_buf_ptr]);
@@ -1391,11 +1415,13 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
int advance = pg_rightmost_one_pos32(mask);
input_buf_ptr += advance;
+ simd_total_advance += advance;
}
else
{
/* No special characters found, so skip the entire chunk */
input_buf_ptr += sizeof(Vector8);
+ simd_total_advance += sizeof(Vector8);
continue;
}
}
@@ -1603,6 +1629,50 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
}
} /* end of outer loop */
+#ifndef USE_NO_SIMD
+
+ /* SIMD was enabled */
+ if (simd_continue)
+ {
+ /* SIMD is worth */
+ if (simd_total_cycle && simd_total_advance / simd_total_cycle >= SIMD_ADVANCE_AT_LEAST)
+ {
+ Assert(cstate->simd_current_sleep_cycle == 0);
+ cstate->simd_last_sleep_cycle >>= 1;
+ }
+ /* SIMD was enabled but it isn't worth */
+ else
+ {
+ uint16 simd_last_sleep_cycle = cstate->simd_last_sleep_cycle;
+
+ cstate->simd_continue = false;
+
+ if (simd_last_sleep_cycle == 0)
+ simd_last_sleep_cycle = 1;
+ else if (simd_last_sleep_cycle >= SIMD_SLEEP_MAX / 2)
+ simd_last_sleep_cycle = SIMD_SLEEP_MAX;
+ else
+ simd_last_sleep_cycle <<= 1;
+ cstate->simd_current_sleep_cycle = simd_last_sleep_cycle;
+ cstate->simd_last_sleep_cycle = simd_last_sleep_cycle;
+ }
+ }
+ /* SIMD was disabled */
+ else
+ {
+ /*
+ * We should come here with decrementing
+ * cstate->simd_current_sleep_cycle from a positive number.
+ */
+ Assert(cstate->simd_current_sleep_cycle != 0);
+ cstate->simd_current_sleep_cycle--;
+
+ if (cstate->simd_current_sleep_cycle == 0)
+ cstate->simd_continue = true;
+ }
+
+#endif
+
/*
* Transfer any still-uncopied data to line_buf.
*/
--
2.51.0
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
@ 2025-10-18 18:46 ` KAZAR Ayoub <ma_kazar@esi.dz>
2025-10-18 20:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2 siblings, 1 reply; 178+ messages in thread
From: KAZAR Ayoub @ 2025-10-18 18:46 UTC (permalink / raw)
To: Nazir Bilal Yavuz <byavuz81@gmail.com>; +Cc: Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
Hello,
I’ve rebenchmarked the new heuristic patch, We still have the previous
improvements ranging from 15% to 30%. For regressions i see at maximum 3%
or 4% in the worst case, so this is solid.
I'm also trying the idea of doing SIMD inside quotes with prefix XOR using
carry less multiplication avoiding the slow path in all cases even with
weird looking input, but it needs to take into consideration the
availability of PCLMULQDQ instruction set with <wmmintrin.h> and here we
go, it quickly starts to become dirty OR we can wait for the decision to
start requiring x86-64-v2 or v3 which has SSE4.2 and AVX2.
Regards,
Ayoub Kazar
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-18 18:46 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
@ 2025-10-18 20:01 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 06:17 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
0 siblings, 1 reply; 178+ messages in thread
From: Nazir Bilal Yavuz @ 2025-10-18 20:01 UTC (permalink / raw)
To: KAZAR Ayoub <ma_kazar@esi.dz>; +Cc: Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
Hi,
On Sat, 18 Oct 2025 at 21:46, KAZAR Ayoub <ma_kazar@esi.dz> wrote:
>
> Hello,
>
> I’ve rebenchmarked the new heuristic patch, We still have the previous improvements ranging from 15% to 30%. For regressions i see at maximum 3% or 4% in the worst case, so this is solid.
Thank you so much for doing this! The results look nice, do you think
there are any other benchmarks that might be interesting to try?
> I'm also trying the idea of doing SIMD inside quotes with prefix XOR using carry less multiplication avoiding the slow path in all cases even with weird looking input, but it needs to take into consideration the availability of PCLMULQDQ instruction set with <wmmintrin.h> and here we go, it quickly starts to become dirty OR we can wait for the decision to start requiring x86-64-v2 or v3 which has SSE4.2 and AVX2.
I can not quite picture this, would you mind sharing a few examples or patches?
--
Regards,
Nazir Bilal Yavuz
Microsoft
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-18 18:46 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-10-18 20:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
@ 2025-10-21 06:17 ` KAZAR Ayoub <ma_kazar@esi.dz>
2025-10-21 06:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-10-21 18:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
0 siblings, 2 replies; 178+ messages in thread
From: KAZAR Ayoub @ 2025-10-21 06:17 UTC (permalink / raw)
To: Nazir Bilal Yavuz <byavuz81@gmail.com>; nathandbossart@gmail.com <nathandbossart@gmail.com>; ants.aasma@cybertec.at <ants.aasma@cybertec.at>; +Cc: Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
On Sat, Oct 18, 2025 at 10:01 PM Nazir Bilal Yavuz <byavuz81@gmail.com>
wrote:
> Thank you so much for doing this! The results look nice, do you think
> there are any other benchmarks that might be interesting to try?
>
> > I'm also trying the idea of doing SIMD inside quotes with prefix XOR
> using carry less multiplication avoiding the slow path in all cases even
> with weird looking input, but it needs to take into consideration the
> availability of PCLMULQDQ instruction set with <wmmintrin.h> and here we
> go, it quickly starts to become dirty OR we can wait for the decision to
> start requiring x86-64-v2 or v3 which has SSE4.2 and AVX2.
>
> I can not quite picture this, would you mind sharing a few examples or
> patches?
>
The idea aims to avoid stopping at characters that are not actually special
in their position (inside quote, escaped ..etc)
This is done by creating a lot of masks from the original chunk, masks
like: quote_mask, escape_mask, odd escape sequences mask ; from these we
can deduce which quotes are not special to stop at
Then for inside quotes, we aim to know which characters in our chunk are
inside quotes (also keeping in track the previous chunk's quote state) and
there's a clever/fast way to do it [1].
After this you start to match with LF and CR ..etc, all this while
maintaining the state of what you've seen (the annoying part).
At the end you only reach the scalar path advancing by the position of
first real special character that requires special treatment.
However, after trying to implement this on the existing pipeline way of
COPY command [2] (broken hopeless try, but has the idea), It becomes very
unreasonable for a lot of reasons:
- It is very challenging to correctly handle commas inside quoted fields,
and tracking quoted vs. unquoted state (especially across chunk boundaries,
or with escaped quotes) ....
- Using carry less multiplication (CLMUL) for prefix xor on a 16 bytes
chunk is overkill for some architectures where PCLMULQDQ latency is high
[3][4] to a point where it performs worse than an unrolled shifts + xor (5
cycles).
- It starts to feel that handling these cases is inherently scalar, doing
all that work for a 16 bytes chunk would be unreasonable since it's not
free, compared to a simple help using SIMD and heuristic of Nazir which is
way nicer in general.
Currently we are at 200-400Mbps which isn't that terrible compared to
production and non production grade parsers (of course we don't only parse
in our case), also we are using SSE2 only so theoretically if we add
support for avx later on we'll have even better numbers.
Maybe more micro optimizations to the current heuristic can squeeze it more.
[1]
https://branchfree.org/2019/03/06/code-fragment-finding-quote-pairs-with-carry-less-multiply-pclmulq...
[2]
https://github.com/AyoubKaz07/postgres/commit/73c6ecfedae4cce5c3f375fd6074b1ca9dfe1daf
[3] https://agner.org/optimize/instruction_tables.pdf
[4] https://www.uops.info/table.html
Regards,
Ayoub Kazar.
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-18 18:46 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-10-18 20:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 06:17 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
@ 2025-10-21 06:44 ` KAZAR Ayoub <ma_kazar@esi.dz>
1 sibling, 0 replies; 178+ messages in thread
From: KAZAR Ayoub @ 2025-10-21 06:44 UTC (permalink / raw)
To: Nazir Bilal Yavuz <byavuz81@gmail.com>; nathandbossart@gmail.com; ants.aasma@cybertec.at; +Cc: Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
On Tue, Oct 21, 2025, 8:17 AM KAZAR Ayoub <ma_kazar@esi.dz> wrote:
>
> Currently we are at 200-400Mbps which isn't that terrible compared to
> production and non production grade parsers (of course we don't only parse
> in our case), also we are using SSE2 only so theoretically if we add
> support for avx later on we'll have even better numbers.
> Maybe more micro optimizations to the current heuristic can squeeze it
> more.
>
>
> [1]
> https://branchfree.org/2019/03/06/code-fragment-finding-quote-pairs-with-carry-less-multiply-pclmulq...
> [2]
> https://github.com/AyoubKaz07/postgres/commit/73c6ecfedae4cce5c3f375fd6074b1ca9dfe1daf
> [3] https://agner.org/optimize/instruction_tables.pdf
> [4] https://www.uops.info/table.html
>
> Regards,
> Ayoub Kazar.
>
Sorry, I meant 200-400MB/s.
Regards.
Ayoub Kazar.
>
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-18 18:46 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-10-18 20:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 06:17 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
@ 2025-10-21 18:55 ` Nathan Bossart <nathandbossart@gmail.com>
1 sibling, 0 replies; 178+ messages in thread
From: Nathan Bossart @ 2025-10-21 18:55 UTC (permalink / raw)
To: KAZAR Ayoub <ma_kazar@esi.dz>; +Cc: Nazir Bilal Yavuz <byavuz81@gmail.com>; ants.aasma@cybertec.at <ants.aasma@cybertec.at>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
On Tue, Oct 21, 2025 at 08:17:01AM +0200, KAZAR Ayoub wrote:
>>> I'm also trying the idea of doing SIMD inside quotes with prefix XOR
>>> using carry less multiplication avoiding the slow path in all cases even
>>> with weird looking input, but it needs to take into consideration the
>>> availability of PCLMULQDQ instruction set with <wmmintrin.h> and here we
>>> go, it quickly starts to become dirty OR we can wait for the decision to
>>> start requiring x86-64-v2 or v3 which has SSE4.2 and AVX2.
>
> [...]
>
> Currently we are at 200-400Mbps which isn't that terrible compared to
> production and non production grade parsers (of course we don't only parse
> in our case), also we are using SSE2 only so theoretically if we add
> support for avx later on we'll have even better numbers.
> Maybe more micro optimizations to the current heuristic can squeeze it more.
I'd greatly prefer that we stick with SSE2/Neon (i.e., simd.h) unless the
gains are extraordinary. Beyond the inherent complexity of using
architecture-specific intrinsics, you also have to deal with configure-time
checks, runtime checks, and function pointer overhead juggling. That tends
to be a lot of work for the amount of gain.
--
nathan
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
@ 2025-10-18 20:01 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2 siblings, 0 replies; 178+ messages in thread
From: Nazir Bilal Yavuz @ 2025-10-18 20:01 UTC (permalink / raw)
To: Andrew Dunstan <andrew@dunslane.net>; +Cc: KAZAR Ayoub <ma_kazar@esi.dz>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
Hi,
On Thu, 16 Oct 2025 at 17:29, Nazir Bilal Yavuz <byavuz81@gmail.com> wrote:
>
> Overall logic is that we will not run SIMD for the entire line and we
> decide if it is worth it to run SIMD for the next lines.
I had a typo there, correct sentence is that:
"Overall logic is that we *will* run SIMD for the entire line and we
decide if it is worth it to run SIMD for the next lines."
--
Regards,
Nazir Bilal Yavuz
Microsoft
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
@ 2025-10-20 14:02 ` Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2 siblings, 1 reply; 178+ messages in thread
From: Andrew Dunstan @ 2025-10-20 14:02 UTC (permalink / raw)
To: Nazir Bilal Yavuz <byavuz81@gmail.com>; +Cc: KAZAR Ayoub <ma_kazar@esi.dz>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
On 2025-10-16 Th 10:29 AM, Nazir Bilal Yavuz wrote:
> Hi,
>
> On Thu, 21 Aug 2025 at 18:47, Andrew Dunstan<andrew@dunslane.net> wrote:
>>
>> On 2025-08-19 Tu 10:14 AM, Nazir Bilal Yavuz wrote:
>>> Hi,
>>>
>>> On Tue, 19 Aug 2025 at 15:33, Nazir Bilal Yavuz<byavuz81@gmail.com> wrote:
>>>> I am able to reproduce the regression you mentioned but both
>>>> regressions are %20 on my end. I found that (by experimenting) SIMD
>>>> causes a regression if it advances less than 5 characters.
>>>>
>>>> So, I implemented a small heuristic. It works like that:
>>>>
>>>> - If advance < 5 -> insert a sleep penalty (n cycles).
>>> 'sleep' might be a poor word choice here. I meant skipping SIMD for n
>>> number of times.
>>>
>> I was thinking a bit about that this morning. I wonder if it might be
>> better instead of having a constantly applied heuristic like this, it
>> might be better to do a little extra accounting in the first, say, 1000
>> lines of an input file, and if less than some portion of the input is
>> found to be special characters then switch to the SIMD code. What that
>> portion should be would need to be determined by some experimentation
>> with a variety of typical workloads, but given your findings 20% seems
>> like a good starting point.
> I implemented a heuristic something similar to this. It is a mix of
> previous heuristic and your idea, it works like that:
>
> Overall logic is that we will not run SIMD for the entire line and we
> decide if it is worth it to run SIMD for the next lines.
>
> 1 - We will try SIMD and decide if it is worth it to run SIMD.
> 1.1 - If it is worth it, we will continue to run SIMD and we will
> halve the simd_last_sleep_cycle variable.
> 1.2 - If it is not worth it, we will double the simd_last_sleep_cycle
> and we will not run SIMD for these many lines.
> 1.3 - After skipping simd_last_sleep_cycle lines, we will go back to the #1.
> Note: simd_last_sleep_cycle can not pass 1024, so we will run SIMD for
> each 1024 lines at max.
>
> With this heuristic the regression is limited by %2 in the worst case.
>
My worry is that the worst case is actually quite common. Sparse data
sets dominated by a lot of null values (and hence lots of special
characters) are very common. Are people prepared to accept a 2%
regression on load times for such data sets?
cheers
andrew
--
Andrew Dunstan
EDB:https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
@ 2025-10-20 17:04 ` Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
0 siblings, 1 reply; 178+ messages in thread
From: Nathan Bossart @ 2025-10-20 17:04 UTC (permalink / raw)
To: Andrew Dunstan <andrew@dunslane.net>; +Cc: Nazir Bilal Yavuz <byavuz81@gmail.com>; KAZAR Ayoub <ma_kazar@esi.dz>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
On Mon, Oct 20, 2025 at 10:02:23AM -0400, Andrew Dunstan wrote:
> On 2025-10-16 Th 10:29 AM, Nazir Bilal Yavuz wrote:
>> With this heuristic the regression is limited by %2 in the worst case.
>
> My worry is that the worst case is actually quite common. Sparse data sets
> dominated by a lot of null values (and hence lots of special characters) are
> very common. Are people prepared to accept a 2% regression on load times for
> such data sets?
Without knowing how common it is, I think it's difficult to judge whether
2% is a reasonable trade-off. If <5% of workloads might see a small
regression while the other >95% see double-digit percentage improvements,
then I might argue that it's fine. But I'm not sure we have any way to
know those sorts of details at the moment.
I'm also at least a little skeptical about the 2% number. IME that's
generally within the noise range and can vary greatly between machines and
test runs.
--
nathan
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
@ 2025-10-20 20:31 ` Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
0 siblings, 1 reply; 178+ messages in thread
From: Andrew Dunstan @ 2025-10-20 20:31 UTC (permalink / raw)
To: Nathan Bossart <nathandbossart@gmail.com>; +Cc: Nazir Bilal Yavuz <byavuz81@gmail.com>; KAZAR Ayoub <ma_kazar@esi.dz>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
On 2025-10-20 Mo 1:04 PM, Nathan Bossart wrote:
> On Mon, Oct 20, 2025 at 10:02:23AM -0400, Andrew Dunstan wrote:
>> On 2025-10-16 Th 10:29 AM, Nazir Bilal Yavuz wrote:
>>> With this heuristic the regression is limited by %2 in the worst case.
>> My worry is that the worst case is actually quite common. Sparse data sets
>> dominated by a lot of null values (and hence lots of special characters) are
>> very common. Are people prepared to accept a 2% regression on load times for
>> such data sets?
> Without knowing how common it is, I think it's difficult to judge whether
> 2% is a reasonable trade-off. If <5% of workloads might see a small
> regression while the other >95% see double-digit percentage improvements,
> then I might argue that it's fine. But I'm not sure we have any way to
> know those sorts of details at the moment.
I guess what I don't understand is why we actually need to do the test
continuously, even using an adaptive algorithm. Data files in my
experience usually have lines with fairly similar shapes. It's highly
unlikely that you will get the the first 1000 (say) lines of a file that
are rich in special characters and then some later significant section
that isn't, or vice versa. Therefore, doing the test once should yield
the correct answer that can be applied to the rest of the file. That
should reduce the worst case regression to ~0% without sacrificing any
of the performance gains. I appreciate the elegance of what Bilal has
done here, but it does seem like overkill.
> I'm also at least a little skeptical about the 2% number. IME that's
> generally within the noise range and can vary greatly between machines and
> test runs.
>
Fair point.
cheers
andrew
--
Andrew Dunstan
EDB:https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
@ 2025-10-20 21:09 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
0 siblings, 1 reply; 178+ messages in thread
From: Nazir Bilal Yavuz @ 2025-10-20 21:09 UTC (permalink / raw)
To: Andrew Dunstan <andrew@dunslane.net>; +Cc: Nathan Bossart <nathandbossart@gmail.com>; KAZAR Ayoub <ma_kazar@esi.dz>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
Hi,
On Mon, 20 Oct 2025 at 23:32, Andrew Dunstan <andrew@dunslane.net> wrote:
>
>
> On 2025-10-20 Mo 1:04 PM, Nathan Bossart wrote:
>
> On Mon, Oct 20, 2025 at 10:02:23AM -0400, Andrew Dunstan wrote:
>
> On 2025-10-16 Th 10:29 AM, Nazir Bilal Yavuz wrote:
>
> With this heuristic the regression is limited by %2 in the worst case.
>
> My worry is that the worst case is actually quite common. Sparse data sets
> dominated by a lot of null values (and hence lots of special characters) are
> very common. Are people prepared to accept a 2% regression on load times for
> such data sets?
>
> Without knowing how common it is, I think it's difficult to judge whether
> 2% is a reasonable trade-off. If <5% of workloads might see a small
> regression while the other >95% see double-digit percentage improvements,
> then I might argue that it's fine. But I'm not sure we have any way to
> know those sorts of details at the moment.
>
>
> I guess what I don't understand is why we actually need to do the test continuously, even using an adaptive algorithm. Data files in my experience usually have lines with fairly similar shapes. It's highly unlikely that you will get the the first 1000 (say) lines of a file that are rich in special characters and then some later significant section that isn't, or vice versa. Therefore, doing the test once should yield the correct answer that can be applied to the rest of the file. That should reduce the worst case regression to ~0% without sacrificing any of the performance gains. I appreciate the elegance of what Bilal has done here, but it does seem like overkill.
I think the problem is deciding how many lines to process before
deciding for the rest. 1000 lines could work for the small sized data
but it might not work for the big sized data. Also, it might cause a
worse regressions for the small sized data. Because of this reason, I
tried to implement a heuristic that will work regardless of the size
of the data. The last heuristic I suggested will run SIMD for
approximately (#number_of_lines / 1024 [1024 is the max number of
lines to sleep before running SIMD again]) lines if all characters in
the data are special characters.
--
Regards,
Nazir Bilal Yavuz
Microsoft
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
@ 2025-10-21 18:40 ` Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
0 siblings, 1 reply; 178+ messages in thread
From: Nathan Bossart @ 2025-10-21 18:40 UTC (permalink / raw)
To: Nazir Bilal Yavuz <byavuz81@gmail.com>; +Cc: Andrew Dunstan <andrew@dunslane.net>; KAZAR Ayoub <ma_kazar@esi.dz>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
On Tue, Oct 21, 2025 at 12:09:27AM +0300, Nazir Bilal Yavuz wrote:
> I think the problem is deciding how many lines to process before
> deciding for the rest. 1000 lines could work for the small sized data
> but it might not work for the big sized data. Also, it might cause a
> worse regressions for the small sized data.
IMHO we have some leeway with smaller amounts of data. If COPY FROM for
1000 rows takes 19 milliseconds as opposed to 11 milliseconds, it seems
unlikely users would be inconvenienced all that much. (Those numbers are
completely made up in order to illustrate my point.)
> Because of this reason, I
> tried to implement a heuristic that will work regardless of the size
> of the data. The last heuristic I suggested will run SIMD for
> approximately (#number_of_lines / 1024 [1024 is the max number of
> lines to sleep before running SIMD again]) lines if all characters in
> the data are special characters.
I wonder if we could mitigate the regression further by spacing out the
checks a bit more. It could be worth comparing a variety of values to
identify what works best with the test data.
--
nathan
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
@ 2025-10-22 12:33 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
0 siblings, 1 reply; 178+ messages in thread
From: Nazir Bilal Yavuz @ 2025-10-22 12:33 UTC (permalink / raw)
To: Nathan Bossart <nathandbossart@gmail.com>; +Cc: Andrew Dunstan <andrew@dunslane.net>; KAZAR Ayoub <ma_kazar@esi.dz>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
Hi,
On Tue, 21 Oct 2025 at 21:40, Nathan Bossart <nathandbossart@gmail.com> wrote:
>
> On Tue, Oct 21, 2025 at 12:09:27AM +0300, Nazir Bilal Yavuz wrote:
> > I think the problem is deciding how many lines to process before
> > deciding for the rest. 1000 lines could work for the small sized data
> > but it might not work for the big sized data. Also, it might cause a
> > worse regressions for the small sized data.
>
> IMHO we have some leeway with smaller amounts of data. If COPY FROM for
> 1000 rows takes 19 milliseconds as opposed to 11 milliseconds, it seems
> unlikely users would be inconvenienced all that much. (Those numbers are
> completely made up in order to illustrate my point.)
>
> > Because of this reason, I
> > tried to implement a heuristic that will work regardless of the size
> > of the data. The last heuristic I suggested will run SIMD for
> > approximately (#number_of_lines / 1024 [1024 is the max number of
> > lines to sleep before running SIMD again]) lines if all characters in
> > the data are special characters.
>
> I wonder if we could mitigate the regression further by spacing out the
> checks a bit more. It could be worth comparing a variety of values to
> identify what works best with the test data.
Do you mean that instead of doubling the SIMD sleep, we should
multiply it by 3 (or another factor)? Or are you referring to
increasing the maximum sleep from 1024? Or possibly both?
--
Regards,
Nazir Bilal Yavuz
Microsoft
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
@ 2025-10-22 19:24 ` Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
0 siblings, 1 reply; 178+ messages in thread
From: Nathan Bossart @ 2025-10-22 19:24 UTC (permalink / raw)
To: Nazir Bilal Yavuz <byavuz81@gmail.com>; +Cc: Andrew Dunstan <andrew@dunslane.net>; KAZAR Ayoub <ma_kazar@esi.dz>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
On Wed, Oct 22, 2025 at 03:33:37PM +0300, Nazir Bilal Yavuz wrote:
> On Tue, 21 Oct 2025 at 21:40, Nathan Bossart <nathandbossart@gmail.com> wrote:
>> I wonder if we could mitigate the regression further by spacing out the
>> checks a bit more. It could be worth comparing a variety of values to
>> identify what works best with the test data.
>
> Do you mean that instead of doubling the SIMD sleep, we should
> multiply it by 3 (or another factor)? Or are you referring to
> increasing the maximum sleep from 1024? Or possibly both?
I'm not sure of the precise details, but the main thrust of my suggestion
is to assume that whatever sampling you do to determine whether to use SIMD
is good for a larger chunk of data. That is, if you are sampling 1K lines
and then using the result to choose whether to use SIMD for the next 100K
lines, we could instead bump the latter number to 1M lines (or something).
That way we minimize the regression for relatively uniform data sets while
retaining some ability to adapt in case things change halfway through a
large table.
--
nathan
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
@ 2025-10-29 22:22 ` Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
0 siblings, 1 reply; 178+ messages in thread
From: Andrew Dunstan @ 2025-10-29 22:22 UTC (permalink / raw)
To: Nathan Bossart <nathandbossart@gmail.com>; Nazir Bilal Yavuz <byavuz81@gmail.com>; +Cc: KAZAR Ayoub <ma_kazar@esi.dz>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
On 2025-10-22 We 3:24 PM, Nathan Bossart wrote:
> On Wed, Oct 22, 2025 at 03:33:37PM +0300, Nazir Bilal Yavuz wrote:
>> On Tue, 21 Oct 2025 at 21:40, Nathan Bossart <nathandbossart@gmail.com> wrote:
>>> I wonder if we could mitigate the regression further by spacing out the
>>> checks a bit more. It could be worth comparing a variety of values to
>>> identify what works best with the test data.
>> Do you mean that instead of doubling the SIMD sleep, we should
>> multiply it by 3 (or another factor)? Or are you referring to
>> increasing the maximum sleep from 1024? Or possibly both?
> I'm not sure of the precise details, but the main thrust of my suggestion
> is to assume that whatever sampling you do to determine whether to use SIMD
> is good for a larger chunk of data. That is, if you are sampling 1K lines
> and then using the result to choose whether to use SIMD for the next 100K
> lines, we could instead bump the latter number to 1M lines (or something).
> That way we minimize the regression for relatively uniform data sets while
> retaining some ability to adapt in case things change halfway through a
> large table.
>
I'd be ok with numbers like this, although I suspect the numbers of
cases where we see shape shifts like this in the middle of a data set
would be vanishingly small.
cheers
andrew
--
Andrew Dunstan
EDB: https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
@ 2025-11-11 22:23 ` Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
0 siblings, 1 reply; 178+ messages in thread
From: Manni Wood @ 2025-11-11 22:23 UTC (permalink / raw)
To: Andrew Dunstan <andrew@dunslane.net>; +Cc: Nathan Bossart <nathandbossart@gmail.com>; Nazir Bilal Yavuz <byavuz81@gmail.com>; KAZAR Ayoub <ma_kazar@esi.dz>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
On Wed, Oct 29, 2025 at 5:23 PM Andrew Dunstan <andrew@dunslane.net> wrote:
>
> On 2025-10-22 We 3:24 PM, Nathan Bossart wrote:
> > On Wed, Oct 22, 2025 at 03:33:37PM +0300, Nazir Bilal Yavuz wrote:
> >> On Tue, 21 Oct 2025 at 21:40, Nathan Bossart <nathandbossart@gmail.com>
> wrote:
> >>> I wonder if we could mitigate the regression further by spacing out the
> >>> checks a bit more. It could be worth comparing a variety of values to
> >>> identify what works best with the test data.
> >> Do you mean that instead of doubling the SIMD sleep, we should
> >> multiply it by 3 (or another factor)? Or are you referring to
> >> increasing the maximum sleep from 1024? Or possibly both?
> > I'm not sure of the precise details, but the main thrust of my suggestion
> > is to assume that whatever sampling you do to determine whether to use
> SIMD
> > is good for a larger chunk of data. That is, if you are sampling 1K
> lines
> > and then using the result to choose whether to use SIMD for the next 100K
> > lines, we could instead bump the latter number to 1M lines (or
> something).
> > That way we minimize the regression for relatively uniform data sets
> while
> > retaining some ability to adapt in case things change halfway through a
> > large table.
> >
>
>
> I'd be ok with numbers like this, although I suspect the numbers of
> cases where we see shape shifts like this in the middle of a data set
> would be vanishingly small.
>
>
> cheers
>
>
> andrew
>
>
> --
> Andrew Dunstan
> EDB: https://www.enterprisedb.com
>
>
>
>
Hello!
I wanted reproduce the results using files attached by Shinya Kato and
Ayoub Kazar. I installed a postgres compiled from master, and then I
installed a postgres built from master plus Nazir Bilal Yavuz's v3 patches
applied.
The master+v3patches postgres naturally performed better on copying into
the database: anywhere from 11% better for the t.csv file produced by
Shinyo's test.sql, to 35% better copying in the t_4096_none.csv file
created by Ayoub Kazar's simd-copy-from-bench.sql.
But here's where it gets weird. The two files created by Ayoub Kazar's
simd-copy-from-bench.sql that are supposed to be slower, t_4096_escape.txt,
and t_4096_quote.csv, actually ran faster on my machine, by 11% and 5%
respectively.
This seems impossible.
A few things I should note:
I timed the commands using the Unix time command, like so:
time psql -X -U mwood -h localhost -d postgres -c '\copy t from
/tmp/t_4096_escape.txt'
For each file, I timed the copy 6 times and took the average.
This was done on my work Linux machine while also running Chrome and an
Open Office spreadsheet; not a dedicated machine only running postgres.
All of the copy results took between 4.5 seconds (Shinyo's t.csv copied
into postgres compiled from master) to 2 seconds (Ayoub
Kazar's t_4096_none.csv copied into postgres compiled from master plus
Nazir's v3 patches).
Perhaps I need to fiddle with the provided SQL to produce larger files to
get longer run times? Maybe sub-second differences won't tell as
interesting a story as minutes-long copy commands?
Thanks for reading this.
--
-- Manni Wood EDB: https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
@ 2025-11-12 14:44 ` KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
0 siblings, 1 reply; 178+ messages in thread
From: KAZAR Ayoub @ 2025-11-12 14:44 UTC (permalink / raw)
To: Manni Wood <manni.wood@enterprisedb.com>; +Cc: Andrew Dunstan <andrew@dunslane.net>; Nathan Bossart <nathandbossart@gmail.com>; Nazir Bilal Yavuz <byavuz81@gmail.com>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
On Tue, Nov 11, 2025 at 11:23 PM Manni Wood <manni.wood@enterprisedb.com>
wrote:
> Hello!
>
> I wanted reproduce the results using files attached by Shinya Kato and
> Ayoub Kazar. I installed a postgres compiled from master, and then I
> installed a postgres built from master plus Nazir Bilal Yavuz's v3 patches
> applied.
>
> The master+v3patches postgres naturally performed better on copying into
> the database: anywhere from 11% better for the t.csv file produced by
> Shinyo's test.sql, to 35% better copying in the t_4096_none.csv file
> created by Ayoub Kazar's simd-copy-from-bench.sql.
>
> But here's where it gets weird. The two files created by Ayoub Kazar's
> simd-copy-from-bench.sql that are supposed to be slower, t_4096_escape.txt,
> and t_4096_quote.csv, actually ran faster on my machine, by 11% and 5%
> respectively.
>
> This seems impossible.
>
> A few things I should note:
>
> I timed the commands using the Unix time command, like so:
>
> time psql -X -U mwood -h localhost -d postgres -c '\copy t from
> /tmp/t_4096_escape.txt'
>
> For each file, I timed the copy 6 times and took the average.
>
> This was done on my work Linux machine while also running Chrome and an
> Open Office spreadsheet; not a dedicated machine only running postgres.
>
Hello,
I think if you do a perf benchmark (if it still reproduces) it would
probably be possible to explain why it's performing like that looking at
the CPI and other metrics and compare it to my findings.
What i also suggest is to make the data close even closer to the worst case
i.e: more special characters where it hurts the switching between SIMD and
scalar processing (in simd-copy-from-bench.sql file), if still does a good
job then there's something to look at.
>
>
> All of the copy results took between 4.5 seconds (Shinyo's t.csv copied
> into postgres compiled from master) to 2 seconds (Ayoub
> Kazar's t_4096_none.csv copied into postgres compiled from master plus
> Nazir's v3 patches).
>
> Perhaps I need to fiddle with the provided SQL to produce larger files to
> get longer run times? Maybe sub-second differences won't tell as
> interesting a story as minutes-long copy commands?
>
I did try it on some GBs (around 2-5GB only), the differences were not that
much, but if you can run this on more GBs (at least 10GB) it would be good
to look at, although i don't suspect anything interesting since the shape
of data is the same for the totality of the COPY.
>
> Thanks for reading this.
> --
> -- Manni Wood EDB: https://www.enterprisedb.com
>
Thanks for the info.
Regards,
Ayoub Kazar.
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
@ 2025-11-13 02:40 ` Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
0 siblings, 1 reply; 178+ messages in thread
From: Manni Wood @ 2025-11-13 02:40 UTC (permalink / raw)
To: KAZAR Ayoub <ma_kazar@esi.dz>; +Cc: Andrew Dunstan <andrew@dunslane.net>; Nathan Bossart <nathandbossart@gmail.com>; Nazir Bilal Yavuz <byavuz81@gmail.com>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
On Wed, Nov 12, 2025 at 8:44 AM KAZAR Ayoub <ma_kazar@esi.dz> wrote:
> On Tue, Nov 11, 2025 at 11:23 PM Manni Wood <manni.wood@enterprisedb.com>
> wrote:
>
>> Hello!
>>
>> I wanted reproduce the results using files attached by Shinya Kato and
>> Ayoub Kazar. I installed a postgres compiled from master, and then I
>> installed a postgres built from master plus Nazir Bilal Yavuz's v3 patches
>> applied.
>>
>> The master+v3patches postgres naturally performed better on copying into
>> the database: anywhere from 11% better for the t.csv file produced by
>> Shinyo's test.sql, to 35% better copying in the t_4096_none.csv file
>> created by Ayoub Kazar's simd-copy-from-bench.sql.
>>
>> But here's where it gets weird. The two files created by Ayoub Kazar's
>> simd-copy-from-bench.sql that are supposed to be slower, t_4096_escape.txt,
>> and t_4096_quote.csv, actually ran faster on my machine, by 11% and 5%
>> respectively.
>>
>> This seems impossible.
>>
>> A few things I should note:
>>
>> I timed the commands using the Unix time command, like so:
>>
>> time psql -X -U mwood -h localhost -d postgres -c '\copy t from
>> /tmp/t_4096_escape.txt'
>>
>> For each file, I timed the copy 6 times and took the average.
>>
>> This was done on my work Linux machine while also running Chrome and an
>> Open Office spreadsheet; not a dedicated machine only running postgres.
>>
> Hello,
> I think if you do a perf benchmark (if it still reproduces) it would
> probably be possible to explain why it's performing like that looking at
> the CPI and other metrics and compare it to my findings.
> What i also suggest is to make the data close even closer to the worst
> case i.e: more special characters where it hurts the switching between SIMD
> and scalar processing (in simd-copy-from-bench.sql file), if still does a
> good job then there's something to look at.
>
>>
>>
>
>> All of the copy results took between 4.5 seconds (Shinyo's t.csv copied
>> into postgres compiled from master) to 2 seconds (Ayoub
>> Kazar's t_4096_none.csv copied into postgres compiled from master plus
>> Nazir's v3 patches).
>>
>> Perhaps I need to fiddle with the provided SQL to produce larger files to
>> get longer run times? Maybe sub-second differences won't tell as
>> interesting a story as minutes-long copy commands?
>>
> I did try it on some GBs (around 2-5GB only), the differences were not
> that much, but if you can run this on more GBs (at least 10GB) it would be
> good to look at, although i don't suspect anything interesting since the
> shape of data is the same for the totality of the COPY.
>
>>
>> Thanks for reading this.
>> --
>> -- Manni Wood EDB: https://www.enterprisedb.com
>>
> Thanks for the info.
>
>
> Regards,
> Ayoub Kazar.
>
Hello again!
It looks like using 10 times the data removed the apparent speedup in the
simd code when the simd code has to deal with t_4096_escape.txt
and t_4096_quote.csv. When both files contain 1,000,000 lines each,
postgres master+v3patch imports 0.63% slower and 0.54% slower respectively.
For 1,000,000 lines of t_4096_none.txt, the v3 patch yields a 30% speedup.
For 1,000,000 lines of t_4096_none.csv, the v3 patch yields a 33% speedup.
I got these numbers just via simple timing, though this time I used psql's
\timing feature. I left psql running rather than launching it each time as
I did when I used the unix "time" command. I ran the copy command 5 times
for each file and averaged the results. Again, this happened on a Linux
machine that also happened to be running Chrome and Open Office's
spreadsheet.
I should probably try to construct some .txt or .csv files that would trip
up the simd on/off heuristic in the v3 patch.
If data "in the wild" tend to be roughly the same "shape" from row to row,
as Andrew's experience has shown, I imagine these million row results bode
well for the v3 patch...
--
-- Manni Wood EDB: https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
@ 2025-11-17 22:16 ` Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 20:42 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
0 siblings, 2 replies; 178+ messages in thread
From: Nathan Bossart @ 2025-11-17 22:16 UTC (permalink / raw)
To: Manni Wood <manni.wood@enterprisedb.com>; +Cc: KAZAR Ayoub <ma_kazar@esi.dz>; Andrew Dunstan <andrew@dunslane.net>; Nazir Bilal Yavuz <byavuz81@gmail.com>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
I'd like to mark myself as the committer this one, but I noticed that the
commitfest entry [0] has been marked as Withdrawn. Could someone either
reopen it or create a new one as appropriate (assuming there is a desire to
continue with it)? I'm hoping to start spending more time on it soon.
[0] https://commitfest.postgresql.org/patch/5952/
--
nathan
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
@ 2025-11-17 22:52 ` Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
1 sibling, 1 reply; 178+ messages in thread
From: Shinya Kato @ 2025-11-17 22:52 UTC (permalink / raw)
To: Nathan Bossart <nathandbossart@gmail.com>; +Cc: Manni Wood <manni.wood@enterprisedb.com>; KAZAR Ayoub <ma_kazar@esi.dz>; Andrew Dunstan <andrew@dunslane.net>; Nazir Bilal Yavuz <byavuz81@gmail.com>; pgsql-hackers
On Tue, Nov 18, 2025, 07:16 Nathan Bossart <nathandbossart@gmail.com> wrote:
> I'd like to mark myself as the committer this one, but I noticed that the
> commitfest entry [0] has been marked as Withdrawn. Could someone either
> reopen it or create a new one as appropriate (assuming there is a desire to
> continue with it)? I'm hoping to start spending more time on it soon.
>
> [0] https://commitfest.postgresql.org/patch/5952/
I closed this entry because I currently don't have enough time to continue
developing this patch. It is fine if someone else reopens it; I will do my
best to see the patch whenever I can.
Shinya
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
@ 2025-11-18 08:04 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
0 siblings, 1 reply; 178+ messages in thread
From: Nazir Bilal Yavuz @ 2025-11-18 08:04 UTC (permalink / raw)
To: Shinya Kato <shinya11.kato@gmail.com>; +Cc: Nathan Bossart <nathandbossart@gmail.com>; Manni Wood <manni.wood@enterprisedb.com>; KAZAR Ayoub <ma_kazar@esi.dz>; Andrew Dunstan <andrew@dunslane.net>; pgsql-hackers
Hi,
On Tue, 18 Nov 2025 at 01:53, Shinya Kato <shinya11.kato@gmail.com> wrote:
>
>
> On Tue, Nov 18, 2025, 07:16 Nathan Bossart <nathandbossart@gmail.com> wrote:
>>
>> I'd like to mark myself as the committer this one, but I noticed that the
>> commitfest entry [0] has been marked as Withdrawn. Could someone either
>> reopen it or create a new one as appropriate (assuming there is a desire to
>> continue with it)? I'm hoping to start spending more time on it soon.
>>
>> [0] https://commitfest.postgresql.org/patch/5952/
>
>
> I closed this entry because I currently don't have enough time to continue developing this patch. It is fine if someone else reopens it; I will do my best to see the patch whenever I can.
Thank you for all your work on this patch.
I would like to continue working on this but I am not sure what are
the correct steps to reopen this commitfest entry. Do I just need to
change commitfest entry's status to 'Needs review'?
--
Regards,
Nazir Bilal Yavuz
Microsoft
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
@ 2025-11-18 14:01 ` Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
0 siblings, 1 reply; 178+ messages in thread
From: Andrew Dunstan @ 2025-11-18 14:01 UTC (permalink / raw)
To: Nazir Bilal Yavuz <byavuz81@gmail.com>; Shinya Kato <shinya11.kato@gmail.com>; +Cc: Nathan Bossart <nathandbossart@gmail.com>; Manni Wood <manni.wood@enterprisedb.com>; KAZAR Ayoub <ma_kazar@esi.dz>; pgsql-hackers
On 2025-11-18 Tu 3:04 AM, Nazir Bilal Yavuz wrote:
> Hi,
>
> On Tue, 18 Nov 2025 at 01:53, Shinya Kato <shinya11.kato@gmail.com> wrote:
>>
>> On Tue, Nov 18, 2025, 07:16 Nathan Bossart <nathandbossart@gmail.com> wrote:
>>> I'd like to mark myself as the committer this one, but I noticed that the
>>> commitfest entry [0] has been marked as Withdrawn. Could someone either
>>> reopen it or create a new one as appropriate (assuming there is a desire to
>>> continue with it)? I'm hoping to start spending more time on it soon.
>>>
>>> [0] https://commitfest.postgresql.org/patch/5952/
>>
>> I closed this entry because I currently don't have enough time to continue developing this patch. It is fine if someone else reopens it; I will do my best to see the patch whenever I can.
> Thank you for all your work on this patch.
>
> I would like to continue working on this but I am not sure what are
> the correct steps to reopen this commitfest entry. Do I just need to
> change commitfest entry's status to 'Needs review'?
That should do it, I believe.
cheers
andrew
--
Andrew Dunstan
EDB: https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
@ 2025-11-18 14:20 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
0 siblings, 1 reply; 178+ messages in thread
From: Nazir Bilal Yavuz @ 2025-11-18 14:20 UTC (permalink / raw)
To: Andrew Dunstan <andrew@dunslane.net>; +Cc: Shinya Kato <shinya11.kato@gmail.com>; Nathan Bossart <nathandbossart@gmail.com>; Manni Wood <manni.wood@enterprisedb.com>; KAZAR Ayoub <ma_kazar@esi.dz>; pgsql-hackers
Hi,
On Tue, 18 Nov 2025 at 17:01, Andrew Dunstan <andrew@dunslane.net> wrote:
>
>
> On 2025-11-18 Tu 3:04 AM, Nazir Bilal Yavuz wrote:
> > Hi,
> >
> > On Tue, 18 Nov 2025 at 01:53, Shinya Kato <shinya11.kato@gmail.com> wrote:
> >>
> >> On Tue, Nov 18, 2025, 07:16 Nathan Bossart <nathandbossart@gmail.com> wrote:
> >>> I'd like to mark myself as the committer this one, but I noticed that the
> >>> commitfest entry [0] has been marked as Withdrawn. Could someone either
> >>> reopen it or create a new one as appropriate (assuming there is a desire to
> >>> continue with it)? I'm hoping to start spending more time on it soon.
> >>>
> >>> [0] https://commitfest.postgresql.org/patch/5952/
> >>
> >> I closed this entry because I currently don't have enough time to continue developing this patch. It is fine if someone else reopens it; I will do my best to see the patch whenever I can.
> > Thank you for all your work on this patch.
> >
> > I would like to continue working on this but I am not sure what are
> > the correct steps to reopen this commitfest entry. Do I just need to
> > change commitfest entry's status to 'Needs review'?
>
> That should do it, I believe.
Thanks, done.
--
Regards,
Nazir Bilal Yavuz
Microsoft
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
@ 2025-11-19 21:01 ` Nathan Bossart <nathandbossart@gmail.com>
2025-11-20 12:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
0 siblings, 2 replies; 178+ messages in thread
From: Nathan Bossart @ 2025-11-19 21:01 UTC (permalink / raw)
To: Nazir Bilal Yavuz <byavuz81@gmail.com>; +Cc: Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; Manni Wood <manni.wood@enterprisedb.com>; KAZAR Ayoub <ma_kazar@esi.dz>; pgsql-hackers
On Tue, Nov 18, 2025 at 05:20:05PM +0300, Nazir Bilal Yavuz wrote:
> Thanks, done.
I took a look at the v3 patches. Here are my high-level thoughts:
+ /*
+ * Parse data and transfer into line_buf. To get benefit from inlining,
+ * call CopyReadLineText() with the constant boolean variables.
+ */
+ if (cstate->simd_continue)
+ result = CopyReadLineText(cstate, is_csv, true);
+ else
+ result = CopyReadLineText(cstate, is_csv, false);
I'm curious whether this actually generates different code, and if it does,
if it's actually faster. We're already branching on cstate->simd_continue
here.
+ /* Load a chunk of data into a vector register */
+ vector8_load(&chunk, (const uint8 *) ©_input_buf[input_buf_ptr]);
In other places, processing 2 or 4 vectors of data at a time has proven
faster. Have you tried that here?
+ /* \n and \r are not special inside quotes */
+ if (!in_quote)
+ match = vector8_or(vector8_eq(chunk, nl), vector8_eq(chunk, cr));
+
+ if (is_csv)
+ {
+ match = vector8_or(match, vector8_eq(chunk, quote));
+ if (escapec != '\0')
+ match = vector8_or(match, vector8_eq(chunk, escape));
+ }
+ else
+ match = vector8_or(match, vector8_eq(chunk, bs));
The amount of branching here catches my eye. Some branching might be
unavoidable, but in general we want to keep these SIMD paths as branch-free
as possible.
+ /*
+ * Found a special character. Advance up to that point and let
+ * the scalar code handle it.
+ */
+ int advance = pg_rightmost_one_pos32(mask);
+
+ input_buf_ptr += advance;
+ simd_total_advance += advance;
Do we actually need to advance here? Or could we just fall through to the
scalar path? My suspicion is that this extra code doesn't gain us much.
+ if (simd_last_sleep_cycle == 0)
+ simd_last_sleep_cycle = 1;
+ else if (simd_last_sleep_cycle >= SIMD_SLEEP_MAX / 2)
+ simd_last_sleep_cycle = SIMD_SLEEP_MAX;
+ else
+ simd_last_sleep_cycle <<= 1;
+ cstate->simd_current_sleep_cycle = simd_last_sleep_cycle;
+ cstate->simd_last_sleep_cycle = simd_last_sleep_cycle;
IMHO we should be looking for ways to simplify this should-we-use-SIMD
code. For example, perhaps we could just disable the SIMD path for 10K or
100K lines any time a special character is found. I'm dubious that a lot
of complexity is warranted.
--
nathan
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
@ 2025-11-20 12:55 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-21 14:48 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-24 21:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
1 sibling, 2 replies; 178+ messages in thread
From: Nazir Bilal Yavuz @ 2025-11-20 12:55 UTC (permalink / raw)
To: Nathan Bossart <nathandbossart@gmail.com>; +Cc: Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; Manni Wood <manni.wood@enterprisedb.com>; KAZAR Ayoub <ma_kazar@esi.dz>; pgsql-hackers
Hi,
Thank you for looking into this!
On Thu, 20 Nov 2025 at 00:01, Nathan Bossart <nathandbossart@gmail.com> wrote:
>
> On Tue, Nov 18, 2025 at 05:20:05PM +0300, Nazir Bilal Yavuz wrote:
> > Thanks, done.
>
> I took a look at the v3 patches. Here are my high-level thoughts:
>
> + /*
> + * Parse data and transfer into line_buf. To get benefit from inlining,
> + * call CopyReadLineText() with the constant boolean variables.
> + */
> + if (cstate->simd_continue)
> + result = CopyReadLineText(cstate, is_csv, true);
> + else
> + result = CopyReadLineText(cstate, is_csv, false);
>
> I'm curious whether this actually generates different code, and if it does,
> if it's actually faster. We're already branching on cstate->simd_continue
> here.
I had the same doubts before but my benchmark shows nice speedup. I
used a test which is full of delimiters. The current code gives 2700
ms but when I changed these lines with the 'result =
CopyReadLineText(cstate, is_csv, cstate->simd_continue);', the result
was 2920 ms. I compiled code with both -O3 and -O2 and the results
were similar.
>
> + /* Load a chunk of data into a vector register */
> + vector8_load(&chunk, (const uint8 *) ©_input_buf[input_buf_ptr]);
>
> In other places, processing 2 or 4 vectors of data at a time has proven
> faster. Have you tried that here?
Sorry, I could not find the related code piece. I only saw the
vector8_load() inside of hex_decode_safe() function and its comment
says:
/*
* We must process 2 vectors at a time since the output will be half the
* length of the input.
*/
But this does not mention any speedup from using 2 vectors at a time.
Could you please show the related code?
>
> + /* \n and \r are not special inside quotes */
> + if (!in_quote)
> + match = vector8_or(vector8_eq(chunk, nl), vector8_eq(chunk, cr));
> +
> + if (is_csv)
> + {
> + match = vector8_or(match, vector8_eq(chunk, quote));
> + if (escapec != '\0')
> + match = vector8_or(match, vector8_eq(chunk, escape));
> + }
> + else
> + match = vector8_or(match, vector8_eq(chunk, bs));
>
> The amount of branching here catches my eye. Some branching might be
> unavoidable, but in general we want to keep these SIMD paths as branch-free
> as possible.
You are right, I will check these branches and will try to remove as
many branches as possible.
>
> + /*
> + * Found a special character. Advance up to that point and let
> + * the scalar code handle it.
> + */
> + int advance = pg_rightmost_one_pos32(mask);
> +
> + input_buf_ptr += advance;
> + simd_total_advance += advance;
>
> Do we actually need to advance here? Or could we just fall through to the
> scalar path? My suspicion is that this extra code doesn't gain us much.
My testing shows that if we advance more than ~5 characters then SIMD
is worth it, but if we advance less than ~5; then code causes a
regression. I used this information while writing a heuristic.
>
> + if (simd_last_sleep_cycle == 0)
> + simd_last_sleep_cycle = 1;
> + else if (simd_last_sleep_cycle >= SIMD_SLEEP_MAX / 2)
> + simd_last_sleep_cycle = SIMD_SLEEP_MAX;
> + else
> + simd_last_sleep_cycle <<= 1;
> + cstate->simd_current_sleep_cycle = simd_last_sleep_cycle;
> + cstate->simd_last_sleep_cycle = simd_last_sleep_cycle;
>
> IMHO we should be looking for ways to simplify this should-we-use-SIMD
> code. For example, perhaps we could just disable the SIMD path for 10K or
> 100K lines any time a special character is found. I'm dubious that a lot
> of complexity is warranted.
I think this is a bit too harsh since SIMD is still worth it if SIMD
can advance more than ~5 character average. I am trying to use SIMD as
much as possible when it is worth it but what you said can remove the
regression completely, perhaps that is the correct way.
--
Regards,
Nazir Bilal Yavuz
Microsoft
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-20 12:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
@ 2025-11-21 14:48 ` Andrew Dunstan <andrew@dunslane.net>
1 sibling, 0 replies; 178+ messages in thread
From: Andrew Dunstan @ 2025-11-21 14:48 UTC (permalink / raw)
To: Nazir Bilal Yavuz <byavuz81@gmail.com>; Nathan Bossart <nathandbossart@gmail.com>; +Cc: Shinya Kato <shinya11.kato@gmail.com>; Manni Wood <manni.wood@enterprisedb.com>; KAZAR Ayoub <ma_kazar@esi.dz>; pgsql-hackers
On 2025-11-20 Th 7:55 AM, Nazir Bilal Yavuz wrote:
> Hi,
>
> Thank you for looking into this!
>
> On Thu, 20 Nov 2025 at 00:01, Nathan Bossart <nathandbossart@gmail.com> wrote:
>
>> IMHO we should be looking for ways to simplify this should-we-use-SIMD
>> code. For example, perhaps we could just disable the SIMD path for 10K or
>> 100K lines any time a special character is found. I'm dubious that a lot
>> of complexity is warranted.
> I think this is a bit too harsh since SIMD is still worth it if SIMD
> can advance more than ~5 character average. I am trying to use SIMD as
> much as possible when it is worth it but what you said can remove the
> regression completely, perhaps that is the correct way.
>
Perhaps a very small regression (say under 1%) in the worst case would
be OK. But the closer you can get that to zero the more acceptable this
will be. Very large loads of sparse data, which will often have lots of
special characters AIUI, are very common, so we should not dismiss the
worst case as an outlier. I still like the idea of testing, say, a
thousand lines every million, or something like that.
cheers
andrew
--
Andrew Dunstan
EDB: https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-20 12:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
@ 2025-11-24 21:59 ` Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 00:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
1 sibling, 1 reply; 178+ messages in thread
From: Nathan Bossart @ 2025-11-24 21:59 UTC (permalink / raw)
To: Nazir Bilal Yavuz <byavuz81@gmail.com>; +Cc: Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; Manni Wood <manni.wood@enterprisedb.com>; KAZAR Ayoub <ma_kazar@esi.dz>; pgsql-hackers
On Thu, Nov 20, 2025 at 03:55:43PM +0300, Nazir Bilal Yavuz wrote:
> On Thu, 20 Nov 2025 at 00:01, Nathan Bossart <nathandbossart@gmail.com> wrote:
>> + /* Load a chunk of data into a vector register */
>> + vector8_load(&chunk, (const uint8 *) ©_input_buf[input_buf_ptr]);
>>
>> In other places, processing 2 or 4 vectors of data at a time has proven
>> faster. Have you tried that here?
>
> Sorry, I could not find the related code piece. I only saw the
> vector8_load() inside of hex_decode_safe() function and its comment
> says:
>
> /*
> * We must process 2 vectors at a time since the output will be half the
> * length of the input.
> */
>
> But this does not mention any speedup from using 2 vectors at a time.
> Could you please show the related code?
See pg_lfind32().
--
nathan
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-20 12:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-24 21:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
@ 2025-11-26 00:09 ` Manni Wood <manni.wood@enterprisedb.com>
0 siblings, 0 replies; 178+ messages in thread
From: Manni Wood @ 2025-11-26 00:09 UTC (permalink / raw)
To: Nathan Bossart <nathandbossart@gmail.com>; +Cc: Nazir Bilal Yavuz <byavuz81@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; KAZAR Ayoub <ma_kazar@esi.dz>; pgsql-hackers
Hello.
I tried Ayoub Kazar's test files again, using Nazir Bilal Yavuz's v3
patches, but with one difference since my last attempt: this time, I used 5
million lines per file. For each 5 million line file, I ran the import 5
times and averaged the results.
(I found that even using 1 million lines could sometimes produce surprising
speedups where the newer algorithm should be at least a tiny bit slower
than the non-simd version.)
The text file with no special characters is 30% faster. The CSV file with
no special characters is 39% faster. The text file with roughly 1/3rd
special characters is 0.5% slower. The CSV file with roughly 1/3rd special
characters is 2.7% slower.
I also tried files that alternated lines with no special characters and
lines with 1/3rd special characters, thinking I could force the algorithm
to continually check whether or not it should use simd and therefore force
more overhead in the try-simd/don't-try-simd housekeeping code. The text
file was still 50% faster. The CSV file was still 13% faster.
On Mon, Nov 24, 2025 at 3:59 PM Nathan Bossart <nathandbossart@gmail.com>
wrote:
> On Thu, Nov 20, 2025 at 03:55:43PM +0300, Nazir Bilal Yavuz wrote:
> > On Thu, 20 Nov 2025 at 00:01, Nathan Bossart <nathandbossart@gmail.com>
> wrote:
> >> + /* Load a chunk of data into a vector register */
> >> + vector8_load(&chunk, (const uint8 *)
> ©_input_buf[input_buf_ptr]);
> >>
> >> In other places, processing 2 or 4 vectors of data at a time has proven
> >> faster. Have you tried that here?
> >
> > Sorry, I could not find the related code piece. I only saw the
> > vector8_load() inside of hex_decode_safe() function and its comment
> > says:
> >
> > /*
> > * We must process 2 vectors at a time since the output will be half the
> > * length of the input.
> > */
> >
> > But this does not mention any speedup from using 2 vectors at a time.
> > Could you please show the related code?
>
> See pg_lfind32().
>
> --
> nathan
>
--
-- Manni Wood EDB: https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
@ 2025-11-26 11:50 ` KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
1 sibling, 1 reply; 178+ messages in thread
From: KAZAR Ayoub @ 2025-11-26 11:50 UTC (permalink / raw)
To: Nathan Bossart <nathandbossart@gmail.com>; +Cc: Nazir Bilal Yavuz <byavuz81@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; Manni Wood <manni.wood@enterprisedb.com>; pgsql-hackers
Hello,
On Wed, Nov 19, 2025 at 10:01 PM Nathan Bossart <nathandbossart@gmail.com>
wrote:
> On Tue, Nov 18, 2025 at 05:20:05PM +0300, Nazir Bilal Yavuz wrote:
> > Thanks, done.
>
> I took a look at the v3 patches. Here are my high-level thoughts:
>
> + /*
> + * Parse data and transfer into line_buf. To get benefit from
> inlining,
> + * call CopyReadLineText() with the constant boolean variables.
> + */
> + if (cstate->simd_continue)
> + result = CopyReadLineText(cstate, is_csv, true);
> + else
> + result = CopyReadLineText(cstate, is_csv, false);
>
> I'm curious whether this actually generates different code, and if it does,
> if it's actually faster. We're already branching on cstate->simd_continue
> here.
I've compiled both versions with -O2 and confirmed they generate different
code. When simd_continue is passed as a constant to CopyReadLineText, the
compiler optimizes out the condition checks from the SIMD path.
A small benchmark on a 1GB+ file shows the expected benefit which is around
6% performance improvement.
I've attached the assembly outputs in case someone wants to check something
else.
Regards,
Ayoub Kazar
Attachments:
[application/octet-stream] copyfromparse-constant.asm (48.0K, ../../CA+K2Rump8NoMRZRZ2r4jHXUJwByasy_c3_b0oaO+TLkSbMD-jw@mail.gmail.com/3-copyfromparse-constant.asm)
download
[application/octet-stream] copyfromparse-variable.asm (47.1K, ../../CA+K2Rump8NoMRZRZ2r4jHXUJwByasy_c3_b0oaO+TLkSbMD-jw@mail.gmail.com/4-copyfromparse-variable.asm)
download
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
@ 2025-11-26 14:21 ` Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
0 siblings, 1 reply; 178+ messages in thread
From: Manni Wood @ 2025-11-26 14:21 UTC (permalink / raw)
To: KAZAR Ayoub <ma_kazar@esi.dz>; +Cc: Nathan Bossart <nathandbossart@gmail.com>; Nazir Bilal Yavuz <byavuz81@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
On Wed, Nov 26, 2025 at 5:51 AM KAZAR Ayoub <ma_kazar@esi.dz> wrote:
> Hello,
> On Wed, Nov 19, 2025 at 10:01 PM Nathan Bossart <nathandbossart@gmail.com>
> wrote:
>
>> On Tue, Nov 18, 2025 at 05:20:05PM +0300, Nazir Bilal Yavuz wrote:
>> > Thanks, done.
>>
>> I took a look at the v3 patches. Here are my high-level thoughts:
>>
>> + /*
>> + * Parse data and transfer into line_buf. To get benefit from
>> inlining,
>> + * call CopyReadLineText() with the constant boolean variables.
>> + */
>> + if (cstate->simd_continue)
>> + result = CopyReadLineText(cstate, is_csv, true);
>> + else
>> + result = CopyReadLineText(cstate, is_csv, false);
>>
>> I'm curious whether this actually generates different code, and if it
>> does,
>> if it's actually faster. We're already branching on cstate->simd_continue
>> here.
>
> I've compiled both versions with -O2 and confirmed they generate different
> code. When simd_continue is passed as a constant to CopyReadLineText, the
> compiler optimizes out the condition checks from the SIMD path.
> A small benchmark on a 1GB+ file shows the expected benefit which is
> around 6% performance improvement.
> I've attached the assembly outputs in case someone wants to check
> something else.
>
>
> Regards,
> Ayoub Kazar
>
Correction to my last post:
I also tried files that alternated lines with no special characters and
lines with 1/3rd special characters, thinking I could force the algorithm
to continually check whether or not it should use simd and therefore force
more overhead in the try-simd/don't-try-simd housekeeping code. The text
file was still 20% faster (not 50% faster as I originally stated --- that
was a typo). The CSV file was still 13% faster.
Also, apologies for posting at the top in my last e-mail.
--
-- Manni Wood EDB: https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
@ 2025-12-06 01:39 ` Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
0 siblings, 1 reply; 178+ messages in thread
From: Manni Wood @ 2025-12-06 01:39 UTC (permalink / raw)
To: KAZAR Ayoub <ma_kazar@esi.dz>; +Cc: Nathan Bossart <nathandbossart@gmail.com>; Nazir Bilal Yavuz <byavuz81@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
On Wed, Nov 26, 2025 at 8:21 AM Manni Wood <manni.wood@enterprisedb.com>
wrote:
>
>
> On Wed, Nov 26, 2025 at 5:51 AM KAZAR Ayoub <ma_kazar@esi.dz> wrote:
>
>> Hello,
>> On Wed, Nov 19, 2025 at 10:01 PM Nathan Bossart <nathandbossart@gmail.com>
>> wrote:
>>
>>> On Tue, Nov 18, 2025 at 05:20:05PM +0300, Nazir Bilal Yavuz wrote:
>>> > Thanks, done.
>>>
>>> I took a look at the v3 patches. Here are my high-level thoughts:
>>>
>>> + /*
>>> + * Parse data and transfer into line_buf. To get benefit from
>>> inlining,
>>> + * call CopyReadLineText() with the constant boolean variables.
>>> + */
>>> + if (cstate->simd_continue)
>>> + result = CopyReadLineText(cstate, is_csv, true);
>>> + else
>>> + result = CopyReadLineText(cstate, is_csv, false);
>>>
>>> I'm curious whether this actually generates different code, and if it
>>> does,
>>> if it's actually faster. We're already branching on
>>> cstate->simd_continue
>>> here.
>>
>> I've compiled both versions with -O2 and confirmed they generate
>> different code. When simd_continue is passed as a constant to
>> CopyReadLineText, the compiler optimizes out the condition checks from the
>> SIMD path.
>> A small benchmark on a 1GB+ file shows the expected benefit which is
>> around 6% performance improvement.
>> I've attached the assembly outputs in case someone wants to check
>> something else.
>>
>>
>> Regards,
>> Ayoub Kazar
>>
>
> Correction to my last post:
>
> I also tried files that alternated lines with no special characters and
> lines with 1/3rd special characters, thinking I could force the algorithm
> to continually check whether or not it should use simd and therefore force
> more overhead in the try-simd/don't-try-simd housekeeping code. The text
> file was still 20% faster (not 50% faster as I originally stated --- that
> was a typo). The CSV file was still 13% faster.
>
> Also, apologies for posting at the top in my last e-mail.
> --
> -- Manni Wood EDB: https://www.enterprisedb.com
>
Hello, all.
Andrew, I tried your suggestion of just reading the first chunk of the copy
file to determine if SIMD is worth using. Attached are v4 versions of the
patches showing a first attempt at doing that.
I attached test.sh.txt to show how I've been testing, with 5 million lines
of the various copy file variations introduced by Ayub Kazar.
The text copy with no special chars is 30% faster. The CSV copy with no
special chars is 48% faster. The text with 1/3rd escapes is 3% slower. The
CSV with 1/3rd quotes is 0.27% slower.
This set of patches follows the simplest suggestion of just testing the
first N lines (actually first N bytes) of the file and then deciding
whether or not to enable SIMD. This set of patches does not follow Andrew's
later suggestion of maybe checking again every million lines or so.
--
-- Manni Wood EDB: https://www.enterprisedb.com
#!/bin/bash
set -e
set -o pipefail
set -u
PG="psql -X -U postgres -h localhost -d postgres"
echo " ======= Create table t"
${PG} <<EOF
DROP TABLE IF EXISTS t;
CREATE UNLOGGED TABLE t (id INT PRIMARY KEY, filler TEXT);
EOF
echo " ======= Text, no special characters; create /tmp/t_none.txt"
${PG} <<EOF
truncate t;
INSERT INTO t
SELECT s, repeat('A', 4096)
FROM generate_series(1, 5000000) AS s;
COPY t TO '/tmp/t_none.txt' (FORMAT text);
EOF
echo " ======= Text, no special characters; load times"
${PG} <<'EOF'
\timing
truncate table t;
\copy t from /tmp/t_none.txt
truncate table t;
\copy t from /tmp/t_none.txt
truncate table t;
\copy t from /tmp/t_none.txt
truncate table t;
\copy t from /tmp/t_none.txt
truncate table t;
\copy t from /tmp/t_none.txt
EOF
rm /tmp/t_none.txt
echo " ======= CSV, no special characters; create /tmp/t_none.csv"
${PG} <<EOF
truncate t;
INSERT INTO t
SELECT s, repeat('A', 4096)
FROM generate_series(1, 5000000) AS s;
COPY t TO '/tmp/t_none.csv' (FORMAT csv, QUOTE '"');
EOF
echo " ======= CSV, no special characters; load times"
${PG} <<'EOF'
\timing
truncate table t;
\copy t from /tmp/t_none.csv (format csv)
truncate table t;
\copy t from /tmp/t_none.csv (format csv)
truncate table t;
\copy t from /tmp/t_none.csv (format csv)
truncate table t;
\copy t from /tmp/t_none.csv (format csv)
truncate table t;
\copy t from /tmp/t_none.csv (format csv)
EOF
rm /tmp/t_none.csv
echo " ======= Text, with 1/3 escapes; create /tmp/t_escape.txt"
${PG} <<'EOF'
truncate t;
INSERT INTO t
SELECT s, repeat('A\A', 1365)
FROM generate_series(1, 5000000) AS s;
COPY t TO '/tmp/t_escape.txt' (FORMAT text);
EOF
echo " ======= Text, with 1/3 escapes; load times"
${PG} <<'EOF'
\timing
truncate table t;
\copy t from /tmp/t_escape.txt
truncate table t;
\copy t from /tmp/t_escape.txt
truncate table t;
\copy t from /tmp/t_escape.txt
truncate table t;
\copy t from /tmp/t_escape.txt
truncate table t;
\copy t from /tmp/t_escape.txt
EOF
rm /tmp/t_escape.txt
echo " ======= CSV, with 1/3 quotes; create /tmp/t_quote.csv"
${PG} <<EOF
truncate t;
INSERT INTO t
SELECT s, repeat('A"A', 1365)
FROM generate_series(1, 5000000) AS s;
COPY t TO '/tmp/t_quote.csv' (FORMAT csv, QUOTE '"');
EOF
echo " ======= CSV, with 1/3 quotes; load times"
${PG} <<'EOF'
\timing
truncate table t;
\copy t from /tmp/t_quote.csv (format csv)
truncate table t;
\copy t from /tmp/t_quote.csv (format csv)
truncate table t;
\copy t from /tmp/t_quote.csv (format csv)
truncate table t;
\copy t from /tmp/t_quote.csv (format csv)
truncate table t;
\copy t from /tmp/t_quote.csv (format csv)
EOF
rm /tmp/t_quote.csv
echo " ======= Drop table t"
${PG} <<EOF
DROP TABLE IF EXISTS t;
EOF
Attachments:
[text/plain] test.sh.txt (2.6K, ../../CAKWEB6oO4gQd+UJBrU=uuUTE8Hv7GMznjMouvn0Lskr52UqjhQ@mail.gmail.com/3-test.sh.txt)
download | inline:
#!/bin/bash
set -e
set -o pipefail
set -u
PG="psql -X -U postgres -h localhost -d postgres"
echo " ======= Create table t"
${PG} <<EOF
DROP TABLE IF EXISTS t;
CREATE UNLOGGED TABLE t (id INT PRIMARY KEY, filler TEXT);
EOF
echo " ======= Text, no special characters; create /tmp/t_none.txt"
${PG} <<EOF
truncate t;
INSERT INTO t
SELECT s, repeat('A', 4096)
FROM generate_series(1, 5000000) AS s;
COPY t TO '/tmp/t_none.txt' (FORMAT text);
EOF
echo " ======= Text, no special characters; load times"
${PG} <<'EOF'
\timing
truncate table t;
\copy t from /tmp/t_none.txt
truncate table t;
\copy t from /tmp/t_none.txt
truncate table t;
\copy t from /tmp/t_none.txt
truncate table t;
\copy t from /tmp/t_none.txt
truncate table t;
\copy t from /tmp/t_none.txt
EOF
rm /tmp/t_none.txt
echo " ======= CSV, no special characters; create /tmp/t_none.csv"
${PG} <<EOF
truncate t;
INSERT INTO t
SELECT s, repeat('A', 4096)
FROM generate_series(1, 5000000) AS s;
COPY t TO '/tmp/t_none.csv' (FORMAT csv, QUOTE '"');
EOF
echo " ======= CSV, no special characters; load times"
${PG} <<'EOF'
\timing
truncate table t;
\copy t from /tmp/t_none.csv (format csv)
truncate table t;
\copy t from /tmp/t_none.csv (format csv)
truncate table t;
\copy t from /tmp/t_none.csv (format csv)
truncate table t;
\copy t from /tmp/t_none.csv (format csv)
truncate table t;
\copy t from /tmp/t_none.csv (format csv)
EOF
rm /tmp/t_none.csv
echo " ======= Text, with 1/3 escapes; create /tmp/t_escape.txt"
${PG} <<'EOF'
truncate t;
INSERT INTO t
SELECT s, repeat('A\A', 1365)
FROM generate_series(1, 5000000) AS s;
COPY t TO '/tmp/t_escape.txt' (FORMAT text);
EOF
echo " ======= Text, with 1/3 escapes; load times"
${PG} <<'EOF'
\timing
truncate table t;
\copy t from /tmp/t_escape.txt
truncate table t;
\copy t from /tmp/t_escape.txt
truncate table t;
\copy t from /tmp/t_escape.txt
truncate table t;
\copy t from /tmp/t_escape.txt
truncate table t;
\copy t from /tmp/t_escape.txt
EOF
rm /tmp/t_escape.txt
echo " ======= CSV, with 1/3 quotes; create /tmp/t_quote.csv"
${PG} <<EOF
truncate t;
INSERT INTO t
SELECT s, repeat('A"A', 1365)
FROM generate_series(1, 5000000) AS s;
COPY t TO '/tmp/t_quote.csv' (FORMAT csv, QUOTE '"');
EOF
echo " ======= CSV, with 1/3 quotes; load times"
${PG} <<'EOF'
\timing
truncate table t;
\copy t from /tmp/t_quote.csv (format csv)
truncate table t;
\copy t from /tmp/t_quote.csv (format csv)
truncate table t;
\copy t from /tmp/t_quote.csv (format csv)
truncate table t;
\copy t from /tmp/t_quote.csv (format csv)
truncate table t;
\copy t from /tmp/t_quote.csv (format csv)
EOF
rm /tmp/t_quote.csv
echo " ======= Drop table t"
${PG} <<EOF
DROP TABLE IF EXISTS t;
EOF
[text/x-patch] v4-0002-Speed-up-COPY-FROM-text-CSV-parsing-using-SIMD.patch (5.0K, ../../CAKWEB6oO4gQd+UJBrU=uuUTE8Hv7GMznjMouvn0Lskr52UqjhQ@mail.gmail.com/4-v4-0002-Speed-up-COPY-FROM-text-CSV-parsing-using-SIMD.patch)
download | inline diff:
From 38b587dda44cb7160ee734cdea55a573f302c3a9 Mon Sep 17 00:00:00 2001
From: Manni Wood <manni.wood@enterprisedb.com>
Date: Fri, 5 Dec 2025 18:33:46 -0600
Subject: [PATCH v4 2/2] Speed up COPY FROM text/CSV parsing using SIMD
Authors: Shinya Kato <shinya11.kato@gmail.com>,
Nazir Bilal Yavuz <byavuz81@gmail.com>,
Ayoub Kazar <ma_kazar@esi.dz>
Reviewers: Andrew Dunstan <andrew@dunslane.net>
Descussion:
https://www.postgresql.org/message-id/flat/CAOzEurSW8cNr6TPKsjrstnPfhf4QyQqB4tnPXGGe8N4e_v7Jig@mail.gmail.com
---
src/backend/commands/copyfrom.c | 3 +++
src/backend/commands/copyfromparse.c | 29 +++++++++++++++++++++++-
src/include/commands/copyfrom_internal.h | 11 +++++++++
3 files changed, 42 insertions(+), 1 deletion(-)
diff --git a/src/backend/commands/copyfrom.c b/src/backend/commands/copyfrom.c
index 12781963b4f..e638623e5b5 100644
--- a/src/backend/commands/copyfrom.c
+++ b/src/backend/commands/copyfrom.c
@@ -1720,6 +1720,9 @@ BeginCopyFrom(ParseState *pstate,
cstate->cur_attname = NULL;
cstate->cur_attval = NULL;
cstate->relname_only = false;
+ cstate->special_chars_encountered = 0;
+ cstate->checked_simd = false;
+ cstate->use_simd = false;
/*
* Allocate buffers for the input pipeline.
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index 1edb525f072..8cfdfcd4cd8 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -1346,6 +1346,28 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
#ifndef USE_NO_SIMD
+ /*
+ * Wait until we have read more than BYTES_PROCESSED_UNTIL_SIMD_CHECK.
+ * cstate->bytes_processed will grow an unpredictable amount with each
+ * call to this function, so just wait until we have crossed the
+ * threshold.
+ */
+ if (!cstate->checked_simd && cstate->bytes_processed > BYTES_PROCESSED_UNTIL_SIMD_CHECK)
+ {
+ cstate->checked_simd = true;
+
+ /*
+ * If we have not read too many special characters
+ * (SPECIAL_CHAR_SIMD_THRESHOLD) then start using SIMD to speed up
+ * processing. This heuristic assumes that input does not vary too
+ * much from line to line and that number of special characters
+ * encountered in the first BYTES_PROCESSED_UNTIL_SIMD_CHECK are
+ * indicitive of the whole file.
+ */
+ if (cstate->special_chars_encountered < SPECIAL_CHAR_SIMD_THRESHOLD)
+ cstate->use_simd = true;
+ }
+
/*
* Use SIMD instructions to efficiently scan the input buffer for
* special characters (e.g., newline, carriage return, quote, and
@@ -1358,7 +1380,7 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
* sequentially. - The remaining buffer is smaller than one vector
* width (sizeof(Vector8)); SIMD operates on fixed-size chunks.
*/
- if (!last_was_esc && copy_buf_len - input_buf_ptr >= sizeof(Vector8))
+ if (cstate->use_simd && !last_was_esc && copy_buf_len - input_buf_ptr >= sizeof(Vector8))
{
Vector8 chunk;
Vector8 match = vector8_broadcast(0);
@@ -1415,6 +1437,7 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
*/
if (c == '\r')
{
+ cstate->special_chars_encountered++;
IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0);
}
@@ -1446,6 +1469,7 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
/* Process \r */
if (c == '\r' && (!is_csv || !in_quote))
{
+ cstate->special_chars_encountered++;
/* Check for \r\n on first line, _and_ handle \r\n. */
if (cstate->eol_type == EOL_UNKNOWN ||
cstate->eol_type == EOL_CRNL)
@@ -1502,6 +1526,7 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
/* Process \n */
if (c == '\n' && (!is_csv || !in_quote))
{
+ cstate->special_chars_encountered++;
if (cstate->eol_type == EOL_CR || cstate->eol_type == EOL_CRNL)
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
@@ -1524,6 +1549,8 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
{
char c2;
+ cstate->special_chars_encountered++;
+
IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0);
IF_NEED_REFILL_AND_EOF_BREAK(0);
diff --git a/src/include/commands/copyfrom_internal.h b/src/include/commands/copyfrom_internal.h
index c8b22af22d8..215215f909f 100644
--- a/src/include/commands/copyfrom_internal.h
+++ b/src/include/commands/copyfrom_internal.h
@@ -181,6 +181,17 @@ typedef struct CopyFromStateData
#define RAW_BUF_BYTES(cstate) ((cstate)->raw_buf_len - (cstate)->raw_buf_index)
uint64 bytes_processed; /* number of bytes processed so far */
+
+ /* the amount of bytes to read until checking if we should try simd */
+#define BYTES_PROCESSED_UNTIL_SIMD_CHECK 100000
+ /* the number of special chars read below which we use simd */
+#define SPECIAL_CHAR_SIMD_THRESHOLD 20000
+ uint64 special_chars_encountered; /* number of special chars
+ * encountered so far */
+ bool checked_simd; /* we read BYTES_PROCESSED_UNTIL_SIMD_CHECK
+ * and checked if we should use SIMD on the
+ * rest of the file */
+ bool use_simd; /* use simd to speed up copying */
} CopyFromStateData;
extern void ReceiveCopyBegin(CopyFromState cstate);
--
2.52.0
[text/x-patch] v4-0001-Speed-up-COPY-FROM-text-CSV-parsing-using-SIMD.patch (3.7K, ../../CAKWEB6oO4gQd+UJBrU=uuUTE8Hv7GMznjMouvn0Lskr52UqjhQ@mail.gmail.com/5-v4-0001-Speed-up-COPY-FROM-text-CSV-parsing-using-SIMD.patch)
download | inline diff:
From 0b1f786bf58c3d90e078d4afa83b7d43dda08491 Mon Sep 17 00:00:00 2001
From: Manni Wood <manni.wood@enterprisedb.com>
Date: Fri, 5 Dec 2025 18:30:00 -0600
Subject: [PATCH v4 1/2] Speed up COPY FROM text/CSV parsing using SIMD
Authors: Shinya Kato <shinya11.kato@gmail.com>,
Nazir Bilal Yavuz <byavuz81@gmail.com>,
Ayoub Kazar <ma_kazar@esi.dz>
Reviewers: Andrew Dunstan <andrew@dunslane.net>
Descussion:
https://www.postgresql.org/message-id/flat/CAOzEurSW8cNr6TPKsjrstnPfhf4QyQqB4tnPXGGe8N4e_v7Jig@mail.gmail.com
---
src/backend/commands/copyfromparse.c | 73 ++++++++++++++++++++++++++++
1 file changed, 73 insertions(+)
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index a09e7fbace3..1edb525f072 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -71,7 +71,9 @@
#include "mb/pg_wchar.h"
#include "miscadmin.h"
#include "pgstat.h"
+#include "port/pg_bitutils.h"
#include "port/pg_bswap.h"
+#include "port/simd.h"
#include "utils/builtins.h"
#include "utils/rel.h"
@@ -1255,6 +1257,14 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
char quotec = '\0';
char escapec = '\0';
+#ifndef USE_NO_SIMD
+ Vector8 nl = vector8_broadcast('\n');
+ Vector8 cr = vector8_broadcast('\r');
+ Vector8 bs = vector8_broadcast('\\');
+ Vector8 quote = vector8_broadcast(0);
+ Vector8 escape = vector8_broadcast(0);
+#endif
+
if (is_csv)
{
quotec = cstate->opts.quote[0];
@@ -1262,6 +1272,12 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
/* ignore special escape processing if it's the same as quotec */
if (quotec == escapec)
escapec = '\0';
+
+#ifndef USE_NO_SIMD
+ quote = vector8_broadcast(quotec);
+ if (quotec != escapec)
+ escape = vector8_broadcast(escapec);
+#endif
}
/*
@@ -1328,6 +1344,63 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
need_data = false;
}
+#ifndef USE_NO_SIMD
+
+ /*
+ * Use SIMD instructions to efficiently scan the input buffer for
+ * special characters (e.g., newline, carriage return, quote, and
+ * escape). This is faster than byte-by-byte iteration, especially on
+ * large buffers.
+ *
+ * We do not apply the SIMD fast path in either of the following
+ * cases: - When the previously processed character was an escape
+ * character (last_was_esc), since the next byte must be examined
+ * sequentially. - The remaining buffer is smaller than one vector
+ * width (sizeof(Vector8)); SIMD operates on fixed-size chunks.
+ */
+ if (!last_was_esc && copy_buf_len - input_buf_ptr >= sizeof(Vector8))
+ {
+ Vector8 chunk;
+ Vector8 match = vector8_broadcast(0);
+ uint32 mask;
+
+ /* Load a chunk of data into a vector register */
+ vector8_load(&chunk, (const uint8 *) ©_input_buf[input_buf_ptr]);
+
+ /* \n and \r are not special inside quotes */
+ if (!in_quote)
+ match = vector8_or(vector8_eq(chunk, nl), vector8_eq(chunk, cr));
+
+ if (is_csv)
+ {
+ match = vector8_or(match, vector8_eq(chunk, quote));
+ if (escapec != '\0')
+ match = vector8_or(match, vector8_eq(chunk, escape));
+ }
+ else
+ match = vector8_or(match, vector8_eq(chunk, bs));
+
+ /* Check if we found any special characters */
+ mask = vector8_highbit_mask(match);
+ if (mask != 0)
+ {
+ /*
+ * Found a special character. Advance up to that point and let
+ * the scalar code handle it.
+ */
+ int advance = pg_rightmost_one_pos32(mask);
+
+ input_buf_ptr += advance;
+ }
+ else
+ {
+ /* No special characters found, so skip the entire chunk */
+ input_buf_ptr += sizeof(Vector8);
+ continue;
+ }
+ }
+#endif
+
/* OK to fetch a character */
prev_raw_ptr = input_buf_ptr;
c = copy_input_buf[input_buf_ptr++];
--
2.52.0
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
@ 2025-12-06 07:55 ` Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
0 siblings, 1 reply; 178+ messages in thread
From: Bilal Yavuz @ 2025-12-06 07:55 UTC (permalink / raw)
To: Manni Wood <manni.wood@enterprisedb.com>; +Cc: KAZAR Ayoub <ma_kazar@esi.dz>; Nathan Bossart <nathandbossart@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
Hi,
On Sat, 6 Dec 2025 at 04:40, Manni Wood <manni.wood@enterprisedb.com> wrote:
> Hello, all.
>
> Andrew, I tried your suggestion of just reading the first chunk of the copy file to determine if SIMD is worth using. Attached are v4 versions of the patches showing a first attempt at doing that.
Thank you for doing this!
> I attached test.sh.txt to show how I've been testing, with 5 million lines of the various copy file variations introduced by Ayub Kazar.
>
> The text copy with no special chars is 30% faster. The CSV copy with no special chars is 48% faster. The text with 1/3rd escapes is 3% slower. The CSV with 1/3rd quotes is 0.27% slower.
>
> This set of patches follows the simplest suggestion of just testing the first N lines (actually first N bytes) of the file and then deciding whether or not to enable SIMD. This set of patches does not follow Andrew's later suggestion of maybe checking again every million lines or so.
My input-generation script is not ready to share yet, but the inputs
follow this format: text_${n}.input, where n represents the number of
normal characters before the delimiter. For example:
n = 0 -> "\n\n\n\n\n..." (no normal characters)
n = 1 -> "a\n..." (1 normal character before the delimiter)
...
n = 5 -> "aaaaa\n..."
… continuing up to n = 32.
Each line has 4096 chars and there are a total of 100000 lines in each
input file.
I only benchmarked the text format. I compared the latest heuristic I
shared [1] with the current method. The benchmarks show roughly a ~16%
regression at the worst case (n = 2), with regressions up to n = 5.
For the remaining values, performance was similar.
Actual comparison of timings (in ms):
current method / heuristic
n = 0 -> 3252.7253 / 2856.2753 (%12)
n = 1 -> 2910.321 / 2520.7717 (%13)
n = 2 -> 2865.008 / 2403.2017 (%16)
n = 3 -> 2608.649 / 2353.1477 (%9)
n = 4 -> 2460.74 / 2300.1783 (%6)
n = 5 -> 2451.696 / 2362.1573 (%3)
No difference for the rest.
Side note: Sorry for the delay in responding, I will continue working
on this next week.
[1] https://postgr.es/m/CAN55FZ1KF7XNpm2XyG%3DM-sFUODai%3D6Z8a11xE3s4YRBeBKY3tA%40mail.gmail.com
--
Regards,
Nazir Bilal Yavuz
Microsoft
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
@ 2025-12-09 13:40 ` Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 22:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-12 20:42 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Mark Wong <markwkm@gmail.com>
0 siblings, 2 replies; 178+ messages in thread
From: Bilal Yavuz @ 2025-12-09 13:40 UTC (permalink / raw)
To: Manni Wood <manni.wood@enterprisedb.com>; +Cc: KAZAR Ayoub <ma_kazar@esi.dz>; Nathan Bossart <nathandbossart@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
Hi,
On Sat, 6 Dec 2025 at 10:55, Bilal Yavuz <byavuz81@gmail.com> wrote:
>
> Hi,
>
> On Sat, 6 Dec 2025 at 04:40, Manni Wood <manni.wood@enterprisedb.com> wrote:
> > Hello, all.
> >
> > Andrew, I tried your suggestion of just reading the first chunk of the copy file to determine if SIMD is worth using. Attached are v4 versions of the patches showing a first attempt at doing that.
>
> Thank you for doing this!
>
> > I attached test.sh.txt to show how I've been testing, with 5 million lines of the various copy file variations introduced by Ayub Kazar.
> >
> > The text copy with no special chars is 30% faster. The CSV copy with no special chars is 48% faster. The text with 1/3rd escapes is 3% slower. The CSV with 1/3rd quotes is 0.27% slower.
> >
> > This set of patches follows the simplest suggestion of just testing the first N lines (actually first N bytes) of the file and then deciding whether or not to enable SIMD. This set of patches does not follow Andrew's later suggestion of maybe checking again every million lines or so.
>
> My input-generation script is not ready to share yet, but the inputs
> follow this format: text_${n}.input, where n represents the number of
> normal characters before the delimiter. For example:
>
> n = 0 -> "\n\n\n\n\n..." (no normal characters)
> n = 1 -> "a\n..." (1 normal character before the delimiter)
> ...
> n = 5 -> "aaaaa\n..."
> … continuing up to n = 32.
>
> Each line has 4096 chars and there are a total of 100000 lines in each
> input file.
>
> I only benchmarked the text format. I compared the latest heuristic I
> shared [1] with the current method. The benchmarks show roughly a ~16%
> regression at the worst case (n = 2), with regressions up to n = 5.
> For the remaining values, performance was similar.
I tried to improve the v4 patchset. My changes are:
1 - I changed CopyReadLineText() to an inline function and sent the
use_simd variable as an argument to get help from inlining.
2 - A main for loop in the CopyReadLineText() function is called many
times, so I moved the use_simd check to the CopyReadLine() function.
3 - Instead of 'bytes_processed', I used 'chars_processed' because
cstate->bytes_processed is increased before we process them and this
can cause wrong results.
4 - Because of #2 and #3, instead of having
'SPECIAL_CHAR_SIMD_THRESHOLD', I used the ratio of 'chars_processed /
special_chars_encountered' to determine whether we want to use SIMD.
5 - cstate->special_chars_encountered is incremented wrongly for the
CSV case. It is not incremented for the quote and escape delimiters. I
moved all increments of cstate->special_chars_encountered to the
central place and tried to optimize it but it still causes a
regression as it creates one more branching.
With these changes, I am able to decrease the regression to %10 from
%16. Regression decreases to %7 if I modify #5 for the only text input
but I did not do that.
My changes are in the 0003.
--
Regards,
Nazir Bilal Yavuz
Microsoft
Attachments:
[text/x-patch] v4.1-0001-Speed-up-COPY-FROM-text-CSV-parsing-using-SIMD.patch (3.5K, ../../CAN55FZ1fwKgGo2wEie1w2M2jzJko6cMi1NWD05Xm47_L9a3D+g@mail.gmail.com/2-v4.1-0001-Speed-up-COPY-FROM-text-CSV-parsing-using-SIMD.patch)
download | inline diff:
From a1b4d28069786c3fb506c79e096312fcfd585fdb Mon Sep 17 00:00:00 2001
From: Shinya Kato <shinya11.kato@gmail.com>
Date: Mon, 28 Jul 2025 22:08:20 +0900
Subject: [PATCH v4.1 1/3] Speed up COPY FROM text/CSV parsing using SIMD
---
src/backend/commands/copyfromparse.c | 76 ++++++++++++++++++++++++++++
1 file changed, 76 insertions(+)
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index 62afcd8fad1..cf110767542 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -71,7 +71,9 @@
#include "mb/pg_wchar.h"
#include "miscadmin.h"
#include "pgstat.h"
+#include "port/pg_bitutils.h"
#include "port/pg_bswap.h"
+#include "port/simd.h"
#include "utils/builtins.h"
#include "utils/rel.h"
@@ -1255,6 +1257,14 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
char quotec = '\0';
char escapec = '\0';
+#ifndef USE_NO_SIMD
+ Vector8 nl = vector8_broadcast('\n');
+ Vector8 cr = vector8_broadcast('\r');
+ Vector8 bs = vector8_broadcast('\\');
+ Vector8 quote = vector8_broadcast(0);
+ Vector8 escape = vector8_broadcast(0);
+#endif
+
if (is_csv)
{
quotec = cstate->opts.quote[0];
@@ -1262,6 +1272,12 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
/* ignore special escape processing if it's the same as quotec */
if (quotec == escapec)
escapec = '\0';
+
+#ifndef USE_NO_SIMD
+ quote = vector8_broadcast(quotec);
+ if (quotec != escapec)
+ escape = vector8_broadcast(escapec);
+#endif
}
/*
@@ -1328,6 +1344,66 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
need_data = false;
}
+#ifndef USE_NO_SIMD
+
+ /*
+ * Use SIMD instructions to efficiently scan the input buffer for
+ * special characters (e.g., newline, carriage return, quote, and
+ * escape). This is faster than byte-by-byte iteration, especially on
+ * large buffers.
+ *
+ * We do not apply the SIMD fast path in either of the following
+ * cases: - When the previously processed character was an escape
+ * character (last_was_esc), since the next byte must be examined
+ * sequentially. - The remaining buffer is smaller than one vector
+ * width (sizeof(Vector8)); SIMD operates on fixed-size chunks.
+ */
+ if (!last_was_esc && copy_buf_len - input_buf_ptr >= sizeof(Vector8))
+ {
+ Vector8 chunk;
+ Vector8 match = vector8_broadcast(0);
+ uint32 mask;
+
+ /* Load a chunk of data into a vector register */
+ vector8_load(&chunk, (const uint8 *) ©_input_buf[input_buf_ptr]);
+
+ if (is_csv)
+ {
+ /* \n and \r are not special inside quotes */
+ if (!in_quote)
+ match = vector8_or(vector8_eq(chunk, nl), vector8_eq(chunk, cr));
+
+ match = vector8_or(match, vector8_eq(chunk, quote));
+ if (escapec != '\0')
+ match = vector8_or(match, vector8_eq(chunk, escape));
+ }
+ else
+ {
+ match = vector8_or(vector8_eq(chunk, nl), vector8_eq(chunk, cr));
+ match = vector8_or(match, vector8_eq(chunk, bs));
+ }
+
+ /* Check if we found any special characters */
+ mask = vector8_highbit_mask(match);
+ if (mask != 0)
+ {
+ /*
+ * Found a special character. Advance up to that point and let
+ * the scalar code handle it.
+ */
+ int advance = pg_rightmost_one_pos32(mask);
+
+ input_buf_ptr += advance;
+ }
+ else
+ {
+ /* No special characters found, so skip the entire chunk */
+ input_buf_ptr += sizeof(Vector8);
+ continue;
+ }
+ }
+#endif
+
/* OK to fetch a character */
prev_raw_ptr = input_buf_ptr;
c = copy_input_buf[input_buf_ptr++];
--
2.51.0
[text/x-patch] v4.1-0002-Speed-up-COPY-FROM-text-CSV-parsing-using-SIMD.patch (5.0K, ../../CAN55FZ1fwKgGo2wEie1w2M2jzJko6cMi1NWD05Xm47_L9a3D+g@mail.gmail.com/3-v4.1-0002-Speed-up-COPY-FROM-text-CSV-parsing-using-SIMD.patch)
download | inline diff:
From 3a2f9ff26755a5248b7a33770f4603fec483d3bc Mon Sep 17 00:00:00 2001
From: Manni Wood <manni.wood@enterprisedb.com>
Date: Fri, 5 Dec 2025 18:33:46 -0600
Subject: [PATCH v4.1 2/3] Speed up COPY FROM text/CSV parsing using SIMD
Authors: Shinya Kato <shinya11.kato@gmail.com>,
Nazir Bilal Yavuz <byavuz81@gmail.com>,
Ayoub Kazar <ma_kazar@esi.dz>
Reviewers: Andrew Dunstan <andrew@dunslane.net>
Descussion:
https://www.postgresql.org/message-id/flat/CAOzEurSW8cNr6TPKsjrstnPfhf4QyQqB4tnPXGGe8N4e_v7Jig@mail.gmail.com
---
src/include/commands/copyfrom_internal.h | 11 +++++++++
src/backend/commands/copyfrom.c | 3 +++
src/backend/commands/copyfromparse.c | 29 +++++++++++++++++++++++-
3 files changed, 42 insertions(+), 1 deletion(-)
diff --git a/src/include/commands/copyfrom_internal.h b/src/include/commands/copyfrom_internal.h
index c8b22af22d8..215215f909f 100644
--- a/src/include/commands/copyfrom_internal.h
+++ b/src/include/commands/copyfrom_internal.h
@@ -181,6 +181,17 @@ typedef struct CopyFromStateData
#define RAW_BUF_BYTES(cstate) ((cstate)->raw_buf_len - (cstate)->raw_buf_index)
uint64 bytes_processed; /* number of bytes processed so far */
+
+ /* the amount of bytes to read until checking if we should try simd */
+#define BYTES_PROCESSED_UNTIL_SIMD_CHECK 100000
+ /* the number of special chars read below which we use simd */
+#define SPECIAL_CHAR_SIMD_THRESHOLD 20000
+ uint64 special_chars_encountered; /* number of special chars
+ * encountered so far */
+ bool checked_simd; /* we read BYTES_PROCESSED_UNTIL_SIMD_CHECK
+ * and checked if we should use SIMD on the
+ * rest of the file */
+ bool use_simd; /* use simd to speed up copying */
} CopyFromStateData;
extern void ReceiveCopyBegin(CopyFromState cstate);
diff --git a/src/backend/commands/copyfrom.c b/src/backend/commands/copyfrom.c
index 12781963b4f..e638623e5b5 100644
--- a/src/backend/commands/copyfrom.c
+++ b/src/backend/commands/copyfrom.c
@@ -1720,6 +1720,9 @@ BeginCopyFrom(ParseState *pstate,
cstate->cur_attname = NULL;
cstate->cur_attval = NULL;
cstate->relname_only = false;
+ cstate->special_chars_encountered = 0;
+ cstate->checked_simd = false;
+ cstate->use_simd = false;
/*
* Allocate buffers for the input pipeline.
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index cf110767542..549b56c21fb 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -1346,6 +1346,28 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
#ifndef USE_NO_SIMD
+ /*
+ * Wait until we have read more than BYTES_PROCESSED_UNTIL_SIMD_CHECK.
+ * cstate->bytes_processed will grow an unpredictable amount with each
+ * call to this function, so just wait until we have crossed the
+ * threshold.
+ */
+ if (!cstate->checked_simd && cstate->bytes_processed > BYTES_PROCESSED_UNTIL_SIMD_CHECK)
+ {
+ cstate->checked_simd = true;
+
+ /*
+ * If we have not read too many special characters
+ * (SPECIAL_CHAR_SIMD_THRESHOLD) then start using SIMD to speed up
+ * processing. This heuristic assumes that input does not vary too
+ * much from line to line and that number of special characters
+ * encountered in the first BYTES_PROCESSED_UNTIL_SIMD_CHECK are
+ * indicitive of the whole file.
+ */
+ if (cstate->special_chars_encountered < SPECIAL_CHAR_SIMD_THRESHOLD)
+ cstate->use_simd = true;
+ }
+
/*
* Use SIMD instructions to efficiently scan the input buffer for
* special characters (e.g., newline, carriage return, quote, and
@@ -1358,7 +1380,7 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
* sequentially. - The remaining buffer is smaller than one vector
* width (sizeof(Vector8)); SIMD operates on fixed-size chunks.
*/
- if (!last_was_esc && copy_buf_len - input_buf_ptr >= sizeof(Vector8))
+ if (cstate->use_simd && !last_was_esc && copy_buf_len - input_buf_ptr >= sizeof(Vector8))
{
Vector8 chunk;
Vector8 match = vector8_broadcast(0);
@@ -1418,6 +1440,7 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
*/
if (c == '\r')
{
+ cstate->special_chars_encountered++;
IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0);
}
@@ -1449,6 +1472,7 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
/* Process \r */
if (c == '\r' && (!is_csv || !in_quote))
{
+ cstate->special_chars_encountered++;
/* Check for \r\n on first line, _and_ handle \r\n. */
if (cstate->eol_type == EOL_UNKNOWN ||
cstate->eol_type == EOL_CRNL)
@@ -1505,6 +1529,7 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
/* Process \n */
if (c == '\n' && (!is_csv || !in_quote))
{
+ cstate->special_chars_encountered++;
if (cstate->eol_type == EOL_CR || cstate->eol_type == EOL_CRNL)
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
@@ -1527,6 +1552,8 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
{
char c2;
+ cstate->special_chars_encountered++;
+
IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0);
IF_NEED_REFILL_AND_EOF_BREAK(0);
--
2.51.0
[text/x-patch] v4.1-0003-Feedback-Changes.patch (7.9K, ../../CAN55FZ1fwKgGo2wEie1w2M2jzJko6cMi1NWD05Xm47_L9a3D+g@mail.gmail.com/4-v4.1-0003-Feedback-Changes.patch)
download | inline diff:
From 8d0e6766175abac15b39884126c29da03657be40 Mon Sep 17 00:00:00 2001
From: Nazir Bilal Yavuz <byavuz81@gmail.com>
Date: Tue, 9 Dec 2025 15:32:10 +0300
Subject: [PATCH v4.1 3/3] Feedback / Changes
---
src/include/commands/copyfrom_internal.h | 9 +--
src/backend/commands/copyfrom.c | 1 +
src/backend/commands/copyfromparse.c | 88 +++++++++++++++---------
3 files changed, 60 insertions(+), 38 deletions(-)
diff --git a/src/include/commands/copyfrom_internal.h b/src/include/commands/copyfrom_internal.h
index 215215f909f..397720bf875 100644
--- a/src/include/commands/copyfrom_internal.h
+++ b/src/include/commands/copyfrom_internal.h
@@ -183,12 +183,13 @@ typedef struct CopyFromStateData
uint64 bytes_processed; /* number of bytes processed so far */
/* the amount of bytes to read until checking if we should try simd */
-#define BYTES_PROCESSED_UNTIL_SIMD_CHECK 100000
- /* the number of special chars read below which we use simd */
-#define SPECIAL_CHAR_SIMD_THRESHOLD 20000
+#define CHARS_PROCESSED_UNTIL_SIMD_CHECK 100000
+ /* the ratio of special chars read below which we use simd */
+#define SPECIAL_CHAR_SIMD_RATIO 4
+ uint64 chars_processed;
uint64 special_chars_encountered; /* number of special chars
* encountered so far */
- bool checked_simd; /* we read BYTES_PROCESSED_UNTIL_SIMD_CHECK
+ bool checked_simd; /* we read CHARS_PROCESSED_UNTIL_SIMD_CHECK
* and checked if we should use SIMD on the
* rest of the file */
bool use_simd; /* use simd to speed up copying */
diff --git a/src/backend/commands/copyfrom.c b/src/backend/commands/copyfrom.c
index e638623e5b5..d44dd16eced 100644
--- a/src/backend/commands/copyfrom.c
+++ b/src/backend/commands/copyfrom.c
@@ -1720,6 +1720,7 @@ BeginCopyFrom(ParseState *pstate,
cstate->cur_attname = NULL;
cstate->cur_attval = NULL;
cstate->relname_only = false;
+ cstate->chars_processed = 0;
cstate->special_chars_encountered = 0;
cstate->checked_simd = false;
cstate->use_simd = false;
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index 549b56c21fb..86a268d0df9 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -143,7 +143,7 @@ static const char BinarySignature[11] = "PGCOPY\n\377\r\n\0";
/* non-export function prototypes */
static bool CopyReadLine(CopyFromState cstate, bool is_csv);
-static bool CopyReadLineText(CopyFromState cstate, bool is_csv);
+static pg_attribute_always_inline bool CopyReadLineText(CopyFromState cstate, bool is_csv, bool use_simd);
static int CopyReadAttributesText(CopyFromState cstate);
static int CopyReadAttributesCSV(CopyFromState cstate);
static Datum CopyReadBinaryAttribute(CopyFromState cstate, FmgrInfo *flinfo,
@@ -1173,8 +1173,40 @@ CopyReadLine(CopyFromState cstate, bool is_csv)
resetStringInfo(&cstate->line_buf);
cstate->line_buf_valid = false;
- /* Parse data and transfer into line_buf */
- result = CopyReadLineText(cstate, is_csv);
+#ifndef USE_NO_SIMD
+
+ /*
+ * Wait until we have read more than CHARS_PROCESSED_UNTIL_SIMD_CHECK.
+ * cstate->bytes_processed will grow an unpredictable amount with each
+ * call to this function, so just wait until we have crossed the
+ * threshold.
+ */
+ if (!cstate->checked_simd && cstate->chars_processed > CHARS_PROCESSED_UNTIL_SIMD_CHECK)
+ {
+ cstate->checked_simd = true;
+
+ /*
+ * If we have not read too many special characters then start using
+ * SIMD to speed up processing. This heuristic assumes that input does
+ * not vary too much from line to line and that number of special
+ * characters encountered in the first
+ * CHARS_PROCESSED_UNTIL_SIMD_CHECK are indicitive of the whole file.
+ */
+ if (cstate->chars_processed / SPECIAL_CHAR_SIMD_RATIO >= cstate->special_chars_encountered)
+ {
+ cstate->use_simd = true;
+ }
+ }
+#endif
+
+ /*
+ * Parse data and transfer into line_buf. To get benefit from inlining,
+ * call CopyReadLineText() with the constant boolean variables.
+ */
+ if (cstate->use_simd)
+ result = CopyReadLineText(cstate, is_csv, true);
+ else
+ result = CopyReadLineText(cstate, is_csv, false);
if (result)
{
@@ -1241,8 +1273,8 @@ CopyReadLine(CopyFromState cstate, bool is_csv)
/*
* CopyReadLineText - inner loop of CopyReadLine for text mode
*/
-static bool
-CopyReadLineText(CopyFromState cstate, bool is_csv)
+static pg_attribute_always_inline bool
+CopyReadLineText(CopyFromState cstate, bool is_csv, bool use_simd)
{
char *copy_input_buf;
int input_buf_ptr;
@@ -1309,7 +1341,7 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
input_buf_ptr = cstate->input_buf_index;
copy_buf_len = cstate->input_buf_len;
- for (;;)
+ for (;; cstate->chars_processed++)
{
int prev_raw_ptr;
char c;
@@ -1346,28 +1378,6 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
#ifndef USE_NO_SIMD
- /*
- * Wait until we have read more than BYTES_PROCESSED_UNTIL_SIMD_CHECK.
- * cstate->bytes_processed will grow an unpredictable amount with each
- * call to this function, so just wait until we have crossed the
- * threshold.
- */
- if (!cstate->checked_simd && cstate->bytes_processed > BYTES_PROCESSED_UNTIL_SIMD_CHECK)
- {
- cstate->checked_simd = true;
-
- /*
- * If we have not read too many special characters
- * (SPECIAL_CHAR_SIMD_THRESHOLD) then start using SIMD to speed up
- * processing. This heuristic assumes that input does not vary too
- * much from line to line and that number of special characters
- * encountered in the first BYTES_PROCESSED_UNTIL_SIMD_CHECK are
- * indicitive of the whole file.
- */
- if (cstate->special_chars_encountered < SPECIAL_CHAR_SIMD_THRESHOLD)
- cstate->use_simd = true;
- }
-
/*
* Use SIMD instructions to efficiently scan the input buffer for
* special characters (e.g., newline, carriage return, quote, and
@@ -1380,7 +1390,7 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
* sequentially. - The remaining buffer is smaller than one vector
* width (sizeof(Vector8)); SIMD operates on fixed-size chunks.
*/
- if (cstate->use_simd && !last_was_esc && copy_buf_len - input_buf_ptr >= sizeof(Vector8))
+ if (use_simd && !last_was_esc && copy_buf_len - input_buf_ptr >= sizeof(Vector8))
{
Vector8 chunk;
Vector8 match = vector8_broadcast(0);
@@ -1430,6 +1440,21 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
prev_raw_ptr = input_buf_ptr;
c = copy_input_buf[input_buf_ptr++];
+ /* Use this calculation decide whether to use SIMD later */
+ if (!use_simd && unlikely(!cstate->checked_simd))
+ {
+ if (is_csv)
+ {
+ if (c == '\r' || c == '\n' || c == quotec || c == escapec)
+ cstate->special_chars_encountered++;
+ }
+ else
+ {
+ if (c == '\r' || c == '\n' || c == '\\')
+ cstate->special_chars_encountered++;
+ }
+ }
+
if (is_csv)
{
/*
@@ -1440,7 +1465,6 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
*/
if (c == '\r')
{
- cstate->special_chars_encountered++;
IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0);
}
@@ -1472,7 +1496,6 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
/* Process \r */
if (c == '\r' && (!is_csv || !in_quote))
{
- cstate->special_chars_encountered++;
/* Check for \r\n on first line, _and_ handle \r\n. */
if (cstate->eol_type == EOL_UNKNOWN ||
cstate->eol_type == EOL_CRNL)
@@ -1529,7 +1552,6 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
/* Process \n */
if (c == '\n' && (!is_csv || !in_quote))
{
- cstate->special_chars_encountered++;
if (cstate->eol_type == EOL_CR || cstate->eol_type == EOL_CRNL)
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
@@ -1552,8 +1574,6 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
{
char c2;
- cstate->special_chars_encountered++;
-
IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0);
IF_NEED_REFILL_AND_EOF_BREAK(0);
--
2.51.0
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
@ 2025-12-09 22:13 ` Manni Wood <manni.wood@enterprisedb.com>
2025-12-10 11:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
1 sibling, 1 reply; 178+ messages in thread
From: Manni Wood @ 2025-12-09 22:13 UTC (permalink / raw)
To: Bilal Yavuz <byavuz81@gmail.com>; +Cc: KAZAR Ayoub <ma_kazar@esi.dz>; Nathan Bossart <nathandbossart@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
On Tue, Dec 9, 2025 at 7:40 AM Bilal Yavuz <byavuz81@gmail.com> wrote:
> Hi,
>
> On Sat, 6 Dec 2025 at 10:55, Bilal Yavuz <byavuz81@gmail.com> wrote:
> >
> > Hi,
> >
> > On Sat, 6 Dec 2025 at 04:40, Manni Wood <manni.wood@enterprisedb.com>
> wrote:
> > > Hello, all.
> > >
> > > Andrew, I tried your suggestion of just reading the first chunk of the
> copy file to determine if SIMD is worth using. Attached are v4 versions of
> the patches showing a first attempt at doing that.
> >
> > Thank you for doing this!
> >
> > > I attached test.sh.txt to show how I've been testing, with 5 million
> lines of the various copy file variations introduced by Ayub Kazar.
> > >
> > > The text copy with no special chars is 30% faster. The CSV copy with
> no special chars is 48% faster. The text with 1/3rd escapes is 3% slower.
> The CSV with 1/3rd quotes is 0.27% slower.
> > >
> > > This set of patches follows the simplest suggestion of just testing
> the first N lines (actually first N bytes) of the file and then deciding
> whether or not to enable SIMD. This set of patches does not follow Andrew's
> later suggestion of maybe checking again every million lines or so.
> >
> > My input-generation script is not ready to share yet, but the inputs
> > follow this format: text_${n}.input, where n represents the number of
> > normal characters before the delimiter. For example:
> >
> > n = 0 -> "\n\n\n\n\n..." (no normal characters)
> > n = 1 -> "a\n..." (1 normal character before the delimiter)
> > ...
> > n = 5 -> "aaaaa\n..."
> > … continuing up to n = 32.
> >
> > Each line has 4096 chars and there are a total of 100000 lines in each
> > input file.
> >
> > I only benchmarked the text format. I compared the latest heuristic I
> > shared [1] with the current method. The benchmarks show roughly a ~16%
> > regression at the worst case (n = 2), with regressions up to n = 5.
> > For the remaining values, performance was similar.
>
> I tried to improve the v4 patchset. My changes are:
>
> 1 - I changed CopyReadLineText() to an inline function and sent the
> use_simd variable as an argument to get help from inlining.
>
> 2 - A main for loop in the CopyReadLineText() function is called many
> times, so I moved the use_simd check to the CopyReadLine() function.
>
> 3 - Instead of 'bytes_processed', I used 'chars_processed' because
> cstate->bytes_processed is increased before we process them and this
> can cause wrong results.
>
> 4 - Because of #2 and #3, instead of having
> 'SPECIAL_CHAR_SIMD_THRESHOLD', I used the ratio of 'chars_processed /
> special_chars_encountered' to determine whether we want to use SIMD.
>
> 5 - cstate->special_chars_encountered is incremented wrongly for the
> CSV case. It is not incremented for the quote and escape delimiters. I
> moved all increments of cstate->special_chars_encountered to the
> central place and tried to optimize it but it still causes a
> regression as it creates one more branching.
>
> With these changes, I am able to decrease the regression to %10 from
> %16. Regression decreases to %7 if I modify #5 for the only text input
> but I did not do that.
>
> My changes are in the 0003.
>
> --
> Regards,
> Nazir Bilal Yavuz
> Microsoft
>
Bilal Yavuz (Nazir Bilal Yavuz?), I did not get a chance to do any work on
this today, but wanted to thank you for finding my logic errors in counting
special chars for CSV, and hacking on my naive solution to make it faster.
By attempting Andrew Dunstan's suggestion, I got a better feel for the
reality that the "housekeeping" code produces a significant amount of
overhead.
--
-- Manni Wood EDB: https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 22:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
@ 2025-12-10 11:59 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-21 20:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Neil Conway <neil.conway@gmail.com>
0 siblings, 1 reply; 178+ messages in thread
From: Nazir Bilal Yavuz @ 2025-12-10 11:59 UTC (permalink / raw)
To: Manni Wood <manni.wood@enterprisedb.com>; +Cc: KAZAR Ayoub <ma_kazar@esi.dz>; Nathan Bossart <nathandbossart@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
Hi,
On Wed, 10 Dec 2025 at 01:13, Manni Wood <manni.wood@enterprisedb.com> wrote:
>
> Bilal Yavuz (Nazir Bilal Yavuz?),
It is Nazir Bilal Yavuz, I changed some settings on my phone and it
seems that it affected my mail account, hopefully it should be fixed
now.
> I did not get a chance to do any work on this today, but wanted to thank you for finding my logic errors in counting special chars for CSV, and hacking on my naive solution to make it faster. By attempting Andrew Dunstan's suggestion, I got a better feel for the reality that the "housekeeping" code produces a significant amount of overhead.
You are welcome! v4.1 has some problems with in_quote case in SIMD
handling code and counting cstate->chars_processed variable. I fixed
them in v4.2.
--
Regards,
Nazir Bilal Yavuz
Microsoft
Attachments:
[text/x-patch] v4.2-0001-Speed-up-COPY-FROM-text-CSV-parsing-using-SIMD.patch (3.7K, ../../CAN55FZ1p5UyUdTRO7iWR_ukjhJDOnpOR2rYNOq=+hcC45OuahQ@mail.gmail.com/2-v4.2-0001-Speed-up-COPY-FROM-text-CSV-parsing-using-SIMD.patch)
download | inline diff:
From e4546b0612bd2fde6190a9ade6e60a1f08299184 Mon Sep 17 00:00:00 2001
From: Manni Wood <manni.wood@enterprisedb.com>
Date: Fri, 5 Dec 2025 18:30:00 -0600
Subject: [PATCH v4.2 1/3] Speed up COPY FROM text/CSV parsing using SIMD
Authors: Shinya Kato <shinya11.kato@gmail.com>,
Nazir Bilal Yavuz <byavuz81@gmail.com>,
Ayoub Kazar <ma_kazar@esi.dz>
Reviewers: Andrew Dunstan <andrew@dunslane.net>
Descussion:
https://www.postgresql.org/message-id/flat/CAOzEurSW8cNr6TPKsjrstnPfhf4QyQqB4tnPXGGe8N4e_v7Jig@mail.gmail.com
---
src/backend/commands/copyfromparse.c | 73 ++++++++++++++++++++++++++++
1 file changed, 73 insertions(+)
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index 62afcd8fad1..673d6683a72 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -71,7 +71,9 @@
#include "mb/pg_wchar.h"
#include "miscadmin.h"
#include "pgstat.h"
+#include "port/pg_bitutils.h"
#include "port/pg_bswap.h"
+#include "port/simd.h"
#include "utils/builtins.h"
#include "utils/rel.h"
@@ -1255,6 +1257,14 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
char quotec = '\0';
char escapec = '\0';
+#ifndef USE_NO_SIMD
+ Vector8 nl = vector8_broadcast('\n');
+ Vector8 cr = vector8_broadcast('\r');
+ Vector8 bs = vector8_broadcast('\\');
+ Vector8 quote = vector8_broadcast(0);
+ Vector8 escape = vector8_broadcast(0);
+#endif
+
if (is_csv)
{
quotec = cstate->opts.quote[0];
@@ -1262,6 +1272,12 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
/* ignore special escape processing if it's the same as quotec */
if (quotec == escapec)
escapec = '\0';
+
+#ifndef USE_NO_SIMD
+ quote = vector8_broadcast(quotec);
+ if (quotec != escapec)
+ escape = vector8_broadcast(escapec);
+#endif
}
/*
@@ -1328,6 +1344,63 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
need_data = false;
}
+#ifndef USE_NO_SIMD
+
+ /*
+ * Use SIMD instructions to efficiently scan the input buffer for
+ * special characters (e.g., newline, carriage return, quote, and
+ * escape). This is faster than byte-by-byte iteration, especially on
+ * large buffers.
+ *
+ * We do not apply the SIMD fast path in either of the following
+ * cases: - When the previously processed character was an escape
+ * character (last_was_esc), since the next byte must be examined
+ * sequentially. - The remaining buffer is smaller than one vector
+ * width (sizeof(Vector8)); SIMD operates on fixed-size chunks.
+ */
+ if (!last_was_esc && copy_buf_len - input_buf_ptr >= sizeof(Vector8))
+ {
+ Vector8 chunk;
+ Vector8 match = vector8_broadcast(0);
+ uint32 mask;
+
+ /* Load a chunk of data into a vector register */
+ vector8_load(&chunk, (const uint8 *) ©_input_buf[input_buf_ptr]);
+
+ /* \n and \r are not special inside quotes */
+ if (!in_quote)
+ match = vector8_or(vector8_eq(chunk, nl), vector8_eq(chunk, cr));
+
+ if (is_csv)
+ {
+ match = vector8_or(match, vector8_eq(chunk, quote));
+ if (escapec != '\0')
+ match = vector8_or(match, vector8_eq(chunk, escape));
+ }
+ else
+ match = vector8_or(match, vector8_eq(chunk, bs));
+
+ /* Check if we found any special characters */
+ mask = vector8_highbit_mask(match);
+ if (mask != 0)
+ {
+ /*
+ * Found a special character. Advance up to that point and let
+ * the scalar code handle it.
+ */
+ int advance = pg_rightmost_one_pos32(mask);
+
+ input_buf_ptr += advance;
+ }
+ else
+ {
+ /* No special characters found, so skip the entire chunk */
+ input_buf_ptr += sizeof(Vector8);
+ continue;
+ }
+ }
+#endif
+
/* OK to fetch a character */
prev_raw_ptr = input_buf_ptr;
c = copy_input_buf[input_buf_ptr++];
--
2.51.0
[text/x-patch] v4.2-0002-Speed-up-COPY-FROM-text-CSV-parsing-using-SIMD.patch (5.0K, ../../CAN55FZ1p5UyUdTRO7iWR_ukjhJDOnpOR2rYNOq=+hcC45OuahQ@mail.gmail.com/3-v4.2-0002-Speed-up-COPY-FROM-text-CSV-parsing-using-SIMD.patch)
download | inline diff:
From 92ac4ada1e4833f81ce30164b48868dc1ade102f Mon Sep 17 00:00:00 2001
From: Manni Wood <manni.wood@enterprisedb.com>
Date: Fri, 5 Dec 2025 18:33:46 -0600
Subject: [PATCH v4.2 2/3] Speed up COPY FROM text/CSV parsing using SIMD
Authors: Shinya Kato <shinya11.kato@gmail.com>,
Nazir Bilal Yavuz <byavuz81@gmail.com>,
Ayoub Kazar <ma_kazar@esi.dz>
Reviewers: Andrew Dunstan <andrew@dunslane.net>
Descussion:
https://www.postgresql.org/message-id/flat/CAOzEurSW8cNr6TPKsjrstnPfhf4QyQqB4tnPXGGe8N4e_v7Jig@mail.gmail.com
---
src/include/commands/copyfrom_internal.h | 11 +++++++++
src/backend/commands/copyfrom.c | 3 +++
src/backend/commands/copyfromparse.c | 29 +++++++++++++++++++++++-
3 files changed, 42 insertions(+), 1 deletion(-)
diff --git a/src/include/commands/copyfrom_internal.h b/src/include/commands/copyfrom_internal.h
index c8b22af22d8..215215f909f 100644
--- a/src/include/commands/copyfrom_internal.h
+++ b/src/include/commands/copyfrom_internal.h
@@ -181,6 +181,17 @@ typedef struct CopyFromStateData
#define RAW_BUF_BYTES(cstate) ((cstate)->raw_buf_len - (cstate)->raw_buf_index)
uint64 bytes_processed; /* number of bytes processed so far */
+
+ /* the amount of bytes to read until checking if we should try simd */
+#define BYTES_PROCESSED_UNTIL_SIMD_CHECK 100000
+ /* the number of special chars read below which we use simd */
+#define SPECIAL_CHAR_SIMD_THRESHOLD 20000
+ uint64 special_chars_encountered; /* number of special chars
+ * encountered so far */
+ bool checked_simd; /* we read BYTES_PROCESSED_UNTIL_SIMD_CHECK
+ * and checked if we should use SIMD on the
+ * rest of the file */
+ bool use_simd; /* use simd to speed up copying */
} CopyFromStateData;
extern void ReceiveCopyBegin(CopyFromState cstate);
diff --git a/src/backend/commands/copyfrom.c b/src/backend/commands/copyfrom.c
index 2ae3d2ba86e..6711c0cfcdd 100644
--- a/src/backend/commands/copyfrom.c
+++ b/src/backend/commands/copyfrom.c
@@ -1720,6 +1720,9 @@ BeginCopyFrom(ParseState *pstate,
cstate->cur_attname = NULL;
cstate->cur_attval = NULL;
cstate->relname_only = false;
+ cstate->special_chars_encountered = 0;
+ cstate->checked_simd = false;
+ cstate->use_simd = false;
/*
* Allocate buffers for the input pipeline.
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index 673d6683a72..d548674c8ff 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -1346,6 +1346,28 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
#ifndef USE_NO_SIMD
+ /*
+ * Wait until we have read more than BYTES_PROCESSED_UNTIL_SIMD_CHECK.
+ * cstate->bytes_processed will grow an unpredictable amount with each
+ * call to this function, so just wait until we have crossed the
+ * threshold.
+ */
+ if (!cstate->checked_simd && cstate->bytes_processed > BYTES_PROCESSED_UNTIL_SIMD_CHECK)
+ {
+ cstate->checked_simd = true;
+
+ /*
+ * If we have not read too many special characters
+ * (SPECIAL_CHAR_SIMD_THRESHOLD) then start using SIMD to speed up
+ * processing. This heuristic assumes that input does not vary too
+ * much from line to line and that number of special characters
+ * encountered in the first BYTES_PROCESSED_UNTIL_SIMD_CHECK are
+ * indicitive of the whole file.
+ */
+ if (cstate->special_chars_encountered < SPECIAL_CHAR_SIMD_THRESHOLD)
+ cstate->use_simd = true;
+ }
+
/*
* Use SIMD instructions to efficiently scan the input buffer for
* special characters (e.g., newline, carriage return, quote, and
@@ -1358,7 +1380,7 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
* sequentially. - The remaining buffer is smaller than one vector
* width (sizeof(Vector8)); SIMD operates on fixed-size chunks.
*/
- if (!last_was_esc && copy_buf_len - input_buf_ptr >= sizeof(Vector8))
+ if (cstate->use_simd && !last_was_esc && copy_buf_len - input_buf_ptr >= sizeof(Vector8))
{
Vector8 chunk;
Vector8 match = vector8_broadcast(0);
@@ -1415,6 +1437,7 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
*/
if (c == '\r')
{
+ cstate->special_chars_encountered++;
IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0);
}
@@ -1446,6 +1469,7 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
/* Process \r */
if (c == '\r' && (!is_csv || !in_quote))
{
+ cstate->special_chars_encountered++;
/* Check for \r\n on first line, _and_ handle \r\n. */
if (cstate->eol_type == EOL_UNKNOWN ||
cstate->eol_type == EOL_CRNL)
@@ -1502,6 +1526,7 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
/* Process \n */
if (c == '\n' && (!is_csv || !in_quote))
{
+ cstate->special_chars_encountered++;
if (cstate->eol_type == EOL_CR || cstate->eol_type == EOL_CRNL)
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
@@ -1524,6 +1549,8 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
{
char c2;
+ cstate->special_chars_encountered++;
+
IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0);
IF_NEED_REFILL_AND_EOF_BREAK(0);
--
2.51.0
[text/x-patch] v4.2-0003-Feedback-Changes.patch (8.5K, ../../CAN55FZ1p5UyUdTRO7iWR_ukjhJDOnpOR2rYNOq=+hcC45OuahQ@mail.gmail.com/4-v4.2-0003-Feedback-Changes.patch)
download | inline diff:
From 128574f80963c5b532c8aa7e7fad84a7e6e20874 Mon Sep 17 00:00:00 2001
From: Nazir Bilal Yavuz <byavuz81@gmail.com>
Date: Tue, 9 Dec 2025 15:32:10 +0300
Subject: [PATCH v4.2 3/3] Feedback / Changes
---
src/include/commands/copyfrom_internal.h | 9 +--
src/backend/commands/copyfrom.c | 1 +
src/backend/commands/copyfromparse.c | 92 +++++++++++++++---------
3 files changed, 65 insertions(+), 37 deletions(-)
diff --git a/src/include/commands/copyfrom_internal.h b/src/include/commands/copyfrom_internal.h
index 215215f909f..397720bf875 100644
--- a/src/include/commands/copyfrom_internal.h
+++ b/src/include/commands/copyfrom_internal.h
@@ -183,12 +183,13 @@ typedef struct CopyFromStateData
uint64 bytes_processed; /* number of bytes processed so far */
/* the amount of bytes to read until checking if we should try simd */
-#define BYTES_PROCESSED_UNTIL_SIMD_CHECK 100000
- /* the number of special chars read below which we use simd */
-#define SPECIAL_CHAR_SIMD_THRESHOLD 20000
+#define CHARS_PROCESSED_UNTIL_SIMD_CHECK 100000
+ /* the ratio of special chars read below which we use simd */
+#define SPECIAL_CHAR_SIMD_RATIO 4
+ uint64 chars_processed;
uint64 special_chars_encountered; /* number of special chars
* encountered so far */
- bool checked_simd; /* we read BYTES_PROCESSED_UNTIL_SIMD_CHECK
+ bool checked_simd; /* we read CHARS_PROCESSED_UNTIL_SIMD_CHECK
* and checked if we should use SIMD on the
* rest of the file */
bool use_simd; /* use simd to speed up copying */
diff --git a/src/backend/commands/copyfrom.c b/src/backend/commands/copyfrom.c
index 6711c0cfcdd..2b77ba2556c 100644
--- a/src/backend/commands/copyfrom.c
+++ b/src/backend/commands/copyfrom.c
@@ -1720,6 +1720,7 @@ BeginCopyFrom(ParseState *pstate,
cstate->cur_attname = NULL;
cstate->cur_attval = NULL;
cstate->relname_only = false;
+ cstate->chars_processed = 0;
cstate->special_chars_encountered = 0;
cstate->checked_simd = false;
cstate->use_simd = false;
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index d548674c8ff..720222152da 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -143,7 +143,7 @@ static const char BinarySignature[11] = "PGCOPY\n\377\r\n\0";
/* non-export function prototypes */
static bool CopyReadLine(CopyFromState cstate, bool is_csv);
-static bool CopyReadLineText(CopyFromState cstate, bool is_csv);
+static pg_attribute_always_inline bool CopyReadLineText(CopyFromState cstate, bool is_csv, bool use_simd);
static int CopyReadAttributesText(CopyFromState cstate);
static int CopyReadAttributesCSV(CopyFromState cstate);
static Datum CopyReadBinaryAttribute(CopyFromState cstate, FmgrInfo *flinfo,
@@ -1173,8 +1173,40 @@ CopyReadLine(CopyFromState cstate, bool is_csv)
resetStringInfo(&cstate->line_buf);
cstate->line_buf_valid = false;
- /* Parse data and transfer into line_buf */
- result = CopyReadLineText(cstate, is_csv);
+#ifndef USE_NO_SIMD
+
+ /*
+ * Wait until we have read more than CHARS_PROCESSED_UNTIL_SIMD_CHECK.
+ * cstate->bytes_processed will grow an unpredictable amount with each
+ * call to this function, so just wait until we have crossed the
+ * threshold.
+ */
+ if (!cstate->checked_simd && cstate->chars_processed > CHARS_PROCESSED_UNTIL_SIMD_CHECK)
+ {
+ cstate->checked_simd = true;
+
+ /*
+ * If we have not read too many special characters then start using
+ * SIMD to speed up processing. This heuristic assumes that input does
+ * not vary too much from line to line and that number of special
+ * characters encountered in the first
+ * CHARS_PROCESSED_UNTIL_SIMD_CHECK are indicitive of the whole file.
+ */
+ if (cstate->chars_processed / SPECIAL_CHAR_SIMD_RATIO >= cstate->special_chars_encountered)
+ {
+ cstate->use_simd = true;
+ }
+ }
+#endif
+
+ /*
+ * Parse data and transfer into line_buf. To get benefit from inlining,
+ * call CopyReadLineText() with the constant boolean variables.
+ */
+ if (cstate->use_simd)
+ result = CopyReadLineText(cstate, is_csv, true);
+ else
+ result = CopyReadLineText(cstate, is_csv, false);
if (result)
{
@@ -1241,11 +1273,12 @@ CopyReadLine(CopyFromState cstate, bool is_csv)
/*
* CopyReadLineText - inner loop of CopyReadLine for text mode
*/
-static bool
-CopyReadLineText(CopyFromState cstate, bool is_csv)
+static pg_attribute_always_inline bool
+CopyReadLineText(CopyFromState cstate, bool is_csv, bool use_simd)
{
char *copy_input_buf;
int input_buf_ptr;
+ int start_input_buf_ptr;
int copy_buf_len;
bool need_data = false;
bool hit_eof = false;
@@ -1309,6 +1342,7 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
input_buf_ptr = cstate->input_buf_index;
copy_buf_len = cstate->input_buf_len;
+ start_input_buf_ptr = input_buf_ptr;
for (;;)
{
int prev_raw_ptr;
@@ -1327,9 +1361,11 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
REFILL_LINEBUF;
CopyLoadInputBuf(cstate);
+ cstate->chars_processed += (input_buf_ptr - start_input_buf_ptr);
/* update our local variables */
hit_eof = cstate->input_reached_eof;
input_buf_ptr = cstate->input_buf_index;
+ start_input_buf_ptr = input_buf_ptr;
copy_buf_len = cstate->input_buf_len;
/*
@@ -1346,28 +1382,6 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
#ifndef USE_NO_SIMD
- /*
- * Wait until we have read more than BYTES_PROCESSED_UNTIL_SIMD_CHECK.
- * cstate->bytes_processed will grow an unpredictable amount with each
- * call to this function, so just wait until we have crossed the
- * threshold.
- */
- if (!cstate->checked_simd && cstate->bytes_processed > BYTES_PROCESSED_UNTIL_SIMD_CHECK)
- {
- cstate->checked_simd = true;
-
- /*
- * If we have not read too many special characters
- * (SPECIAL_CHAR_SIMD_THRESHOLD) then start using SIMD to speed up
- * processing. This heuristic assumes that input does not vary too
- * much from line to line and that number of special characters
- * encountered in the first BYTES_PROCESSED_UNTIL_SIMD_CHECK are
- * indicitive of the whole file.
- */
- if (cstate->special_chars_encountered < SPECIAL_CHAR_SIMD_THRESHOLD)
- cstate->use_simd = true;
- }
-
/*
* Use SIMD instructions to efficiently scan the input buffer for
* special characters (e.g., newline, carriage return, quote, and
@@ -1380,7 +1394,7 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
* sequentially. - The remaining buffer is smaller than one vector
* width (sizeof(Vector8)); SIMD operates on fixed-size chunks.
*/
- if (cstate->use_simd && !last_was_esc && copy_buf_len - input_buf_ptr >= sizeof(Vector8))
+ if (use_simd && !last_was_esc && copy_buf_len - input_buf_ptr >= sizeof(Vector8))
{
Vector8 chunk;
Vector8 match = vector8_broadcast(0);
@@ -1427,6 +1441,21 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
prev_raw_ptr = input_buf_ptr;
c = copy_input_buf[input_buf_ptr++];
+ /* Use this calculation decide whether to use SIMD later */
+ if (!use_simd && unlikely(!cstate->checked_simd))
+ {
+ if (is_csv)
+ {
+ if (c == '\r' || c == '\n' || c == quotec || c == escapec)
+ cstate->special_chars_encountered++;
+ }
+ else
+ {
+ if (c == '\r' || c == '\n' || c == '\\')
+ cstate->special_chars_encountered++;
+ }
+ }
+
if (is_csv)
{
/*
@@ -1437,7 +1466,6 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
*/
if (c == '\r')
{
- cstate->special_chars_encountered++;
IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0);
}
@@ -1469,7 +1497,6 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
/* Process \r */
if (c == '\r' && (!is_csv || !in_quote))
{
- cstate->special_chars_encountered++;
/* Check for \r\n on first line, _and_ handle \r\n. */
if (cstate->eol_type == EOL_UNKNOWN ||
cstate->eol_type == EOL_CRNL)
@@ -1526,7 +1553,6 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
/* Process \n */
if (c == '\n' && (!is_csv || !in_quote))
{
- cstate->special_chars_encountered++;
if (cstate->eol_type == EOL_CR || cstate->eol_type == EOL_CRNL)
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
@@ -1549,8 +1575,6 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
{
char c2;
- cstate->special_chars_encountered++;
-
IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0);
IF_NEED_REFILL_AND_EOF_BREAK(0);
@@ -1635,6 +1659,8 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
*/
REFILL_LINEBUF;
+ cstate->chars_processed += (input_buf_ptr - start_input_buf_ptr);
+
return result;
}
--
2.51.0
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 22:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-10 11:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
@ 2026-01-21 20:49 ` Neil Conway <neil.conway@gmail.com>
2026-01-22 19:32 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-01-31 16:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
0 siblings, 2 replies; 178+ messages in thread
From: Neil Conway @ 2026-01-21 20:49 UTC (permalink / raw)
To: Nazir Bilal Yavuz <byavuz81@gmail.com>; +Cc: Manni Wood <manni.wood@enterprisedb.com>; KAZAR Ayoub <ma_kazar@esi.dz>; Nathan Bossart <nathandbossart@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
A few suggestions:
* I'm curious if we'll see better performance on large inputs if we flush
to `line_buf` periodically (e.g., at least every few thousand bytes or so).
Otherwise we might see poor data cache behavior if large inputs with no
control characters get evicted before we've copied them over. See the
approach taken in escape_json_with_len() in utils/adt/json.c
* Did you compare the approach taken in the patch with a simpler approach
that just does
if (!(vector8_has(chunk, '\\') ||
vector8_has(chunk, '\r') ||
vector8_has(chunk, '\n') /* and so on, accounting for CSV / escapec /
quotec stuff */))
{
/* skip chunk */
}
That's roughly what we do elsewhere (e.g., escape_json_with_len). It has
the advantage of being more readable, along with potentially having fewer
data dependencies.
Neil
On Wed, Dec 10, 2025 at 7:00 AM Nazir Bilal Yavuz <byavuz81@gmail.com>
wrote:
> Hi,
>
> On Wed, 10 Dec 2025 at 01:13, Manni Wood <manni.wood@enterprisedb.com>
> wrote:
> >
> > Bilal Yavuz (Nazir Bilal Yavuz?),
>
> It is Nazir Bilal Yavuz, I changed some settings on my phone and it
> seems that it affected my mail account, hopefully it should be fixed
> now.
>
> > I did not get a chance to do any work on this today, but wanted to thank
> you for finding my logic errors in counting special chars for CSV, and
> hacking on my naive solution to make it faster. By attempting Andrew
> Dunstan's suggestion, I got a better feel for the reality that the
> "housekeeping" code produces a significant amount of overhead.
>
> You are welcome! v4.1 has some problems with in_quote case in SIMD
> handling code and counting cstate->chars_processed variable. I fixed
> them in v4.2.
>
> --
> Regards,
> Nazir Bilal Yavuz
> Microsoft
>
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 22:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-10 11:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-21 20:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Neil Conway <neil.conway@gmail.com>
@ 2026-01-22 19:32 ` KAZAR Ayoub <ma_kazar@esi.dz>
1 sibling, 0 replies; 178+ messages in thread
From: KAZAR Ayoub @ 2026-01-22 19:32 UTC (permalink / raw)
To: Neil Conway <neil.conway@gmail.com>; +Cc: Nazir Bilal Yavuz <byavuz81@gmail.com>; Manni Wood <manni.wood@enterprisedb.com>; Nathan Bossart <nathandbossart@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
On Wed, Jan 21, 2026 at 9:50 PM Neil Conway <neil.conway@gmail.com> wrote:
> A few suggestions:
>
> * I'm curious if we'll see better performance on large inputs if we flush
> to `line_buf` periodically (e.g., at least every few thousand bytes or so).
> Otherwise we might see poor data cache behavior if large inputs with no
> control characters get evicted before we've copied them over. See the
> approach taken in escape_json_with_len() in utils/adt/json.c
>
Hello Neil, thanks for the suggestions!
This might be true as the json code considers it, i'll take a look later.
>
> * Did you compare the approach taken in the patch with a simpler approach
> that just does
>
> if (!(vector8_has(chunk, '\\') ||
> vector8_has(chunk, '\r') ||
> vector8_has(chunk, '\n') /* and so on, accounting for CSV / escapec
> / quotec stuff */))
> {
> /* skip chunk */
> }
>
> That's roughly what we do elsewhere (e.g., escape_json_with_len). It has
> the advantage of being more readable, along with potentially having fewer
> data dependencies.
>
This is true and also similar to the concerns of the very first patches
where i tried to just replace the special chars detection by something less
dependent but I missed vector8_has.
I have tried this now and here are my thoughts and results:
v5: the simplest of all, we take a chunk, detect any special character, if
it's clean then skip it all, otherwise fallback to scalar WITHOUT advancing
to the first special char found, this is mentioned just for reference and
comparison.
Numbers compared to master:
text, no special: +16.8%
text, 1/3 special: -21.3%
csv, no special: +29.5%
csv, 1/3 special: -33.5%
v5.1: Take a chunk, if its clean of specials, skip it, if not process the
whole chunk byte-by-byte, retry the same thing for next chunk if it's fully
skippable and so on...
Numbers:
text, no special: +16.5%
text, 1/3 special: -20.1% (weird) ; but overall the idea is clear maybe.
csv, no special: +29.4%
csv, 1/3 special: -4.4%
For files of 1/3 specials it's almost always terrible to consider simd
because it will always fall back to scalar without even advancing to the
first special character, even for something less than 1/3 i think it would
be same case, i attached an extremely simple analysis on probability of
having a 16 byte clean chunk depending on data we have.
Attached also is an overall comparison graph between master, v3, v5.1 (with
patch)
Regards,
Ayoub
Attachments:
[image/png] proba-spec-chars-chunk.png (42.3K, ../../CA+K2Rumkq2biMOaS3XD_x2M6vwy+yPpraeo3oMvarjVw5OJfiw@mail.gmail.com/3-proba-spec-chars-chunk.png)
download | view image
[image/png] copy-variants-versions.png (17.1K, ../../CA+K2Rumkq2biMOaS3XD_x2M6vwy+yPpraeo3oMvarjVw5OJfiw@mail.gmail.com/4-copy-variants-versions.png)
download | view image
[text/x-patch] v5.1-0001-Simple-heuristic-for-SIMD-COPY-FROM.patch (2.4K, ../../CA+K2Rumkq2biMOaS3XD_x2M6vwy+yPpraeo3oMvarjVw5OJfiw@mail.gmail.com/5-v5.1-0001-Simple-heuristic-for-SIMD-COPY-FROM.patch)
download | inline diff:
From 40cd491a8dcc2b90615ba76f7c4b16e4491fcd04 Mon Sep 17 00:00:00 2001
From: AyoubKAZ <kazarayoub2004@gmail.com>
Date: Thu, 22 Jan 2026 20:02:51 +0100
Subject: [PATCH] Simple heuristic for SIMD COPY FROM
---
src/backend/commands/copyfromparse.c | 32 ++++++++++++++++++++++++++++
1 file changed, 32 insertions(+)
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index 5868a7fa11f..d138dd3731c 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -71,7 +71,9 @@
#include "mb/pg_wchar.h"
#include "miscadmin.h"
#include "pgstat.h"
+#include "port/pg_bitutils.h"
#include "port/pg_bswap.h"
+#include "port/simd.h"
#include "utils/builtins.h"
#include "utils/rel.h"
@@ -1248,6 +1250,7 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
bool need_data = false;
bool hit_eof = false;
bool result = false;
+ int simd_boundary;
/* CSV variables */
bool in_quote = false,
@@ -1292,6 +1295,7 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
copy_input_buf = cstate->input_buf;
input_buf_ptr = cstate->input_buf_index;
copy_buf_len = cstate->input_buf_len;
+ simd_boundary = input_buf_ptr;
for (;;)
{
@@ -1315,6 +1319,7 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
hit_eof = cstate->input_reached_eof;
input_buf_ptr = cstate->input_buf_index;
copy_buf_len = cstate->input_buf_len;
+ simd_boundary = input_buf_ptr;
/*
* If we are completely out of data, break out of the loop,
@@ -1328,6 +1333,33 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
need_data = false;
}
+
+#ifndef USE_NO_SIMD
+
+ if (input_buf_ptr >= simd_boundary && !last_was_esc && (copy_buf_len - input_buf_ptr) >= sizeof(Vector8))
+ {
+ Vector8 chunk;
+
+ vector8_load(&chunk, (const uint8 *) ©_input_buf[input_buf_ptr]);
+
+ if (!(vector8_has(chunk, '\\') ||
+ vector8_has(chunk, '\r') ||
+ vector8_has(chunk, '\n') ||
+ (is_csv && vector8_has(chunk, quotec)) ||
+ (is_csv && escapec != '\0' && vector8_has(chunk, escapec))))
+ {
+ input_buf_ptr += sizeof(Vector8);
+ simd_boundary = input_buf_ptr;
+ continue;
+ }
+ else
+ {
+
+ simd_boundary = input_buf_ptr + sizeof(Vector8);
+ }
+ }
+#endif
+
/* OK to fetch a character */
prev_raw_ptr = input_buf_ptr;
c = copy_input_buf[input_buf_ptr++];
--
2.34.1
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 22:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-10 11:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-21 20:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Neil Conway <neil.conway@gmail.com>
@ 2026-01-31 16:20 ` KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-02 14:17 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-06 13:51 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
1 sibling, 2 replies; 178+ messages in thread
From: KAZAR Ayoub @ 2026-01-31 16:20 UTC (permalink / raw)
To: Neil Conway <neil.conway@gmail.com>; +Cc: Nazir Bilal Yavuz <byavuz81@gmail.com>; Manni Wood <manni.wood@enterprisedb.com>; Nathan Bossart <nathandbossart@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
Hello,
On Wed, Jan 21, 2026 at 9:50 PM Neil Conway <neil.conway@gmail.com> wrote:
> A few suggestions:
>
> * I'm curious if we'll see better performance on large inputs if we flush
> to `line_buf` periodically (e.g., at least every few thousand bytes or so).
> Otherwise we might see poor data cache behavior if large inputs with no
> control characters get evicted before we've copied them over. See the
> approach taken in escape_json_with_len() in utils/adt/json.c
>
> So i gave this a try, attached is the small patch that has v3 + the
suggestion added, here are the results with different threshold for
line_buf refill:
Execution time compared to master:
Workload v3 v3.1 (2k) v3.1 (4k) v3.1 (8k) v3.1 (16k) v3.1 (20k) v3.1 (28k)
text/none -16.5% -17.4% -14.3% -12.6% -13.6% -10.5% -16.3%
text/esc +5.6% +11.1% +3.1% +7.6% +3.0% +4.9% +4.2%
csv/none -31.0% -29.9% -26.7% -30.1% -27.9% -30.2% -29.6%
csv/quote +0.2% -0.6% -0.4% -1.0% +0.1% +2.5% -1.0%
L1d cache miss rates:
Workload Master v3 v3.1 (2k) v3.1 (4k) v3.1 (8k) v3.1 (16k) v3.1 (20k) v3.1
(28k)
text/none 0.20% 0.23% 0.21% 0.22% 0.21% 0.21% 0.21% 0.22%
text/esc 0.21% 0.22% 0.22% 0.22% 0.22% 0.21% 0.22% 0.22%
csv/none 0.17% 0.22% 0.21% 0.22% 0.21% 0.21% 0.22% 0.22%
csv/quote 0.18% 0.22% 0.19% 0.20% 0.20% 0.19% 0.20% 0.20%
On my laptop I have 32KB L1 cache per core.
Results are super close, it is hard to see in the cache misses numbers but
execution times are saying other things, doing the periodic filling of
line_buf seems good to do.
If Manni can rerun the benchmarks on these too, it would be nice to confirm
this.
Regards,
Ayoub
Attachments:
[application/x-patch] 0001-COPY-from-SIMD-v3-with-line_buf-periodic-refill.patch (8.5K, ../../CA+K2Ru=C_woAnd-3-pGHoNSTR8FOf=7eeSWE1xaLt9ojVWndVg@mail.gmail.com/3-0001-COPY-from-SIMD-v3-with-line_buf-periodic-refill.patch)
download | inline diff:
From 40230259b9c867352094d868f0f34cc6fd418c15 Mon Sep 17 00:00:00 2001
From: AyoubKAZ <kazarayoub2004@gmail.com>
Date: Sat, 31 Jan 2026 16:23:35 +0100
Subject: [PATCH] COPY from SIMD v3 with line_buf periodic refill
---
src/backend/commands/copyfrom.c | 6 +
src/backend/commands/copyfromparse.c | 162 ++++++++++++++++++++++-
src/include/commands/copyfrom_internal.h | 7 +
3 files changed, 170 insertions(+), 5 deletions(-)
diff --git a/src/backend/commands/copyfrom.c b/src/backend/commands/copyfrom.c
index 25ee20b23db..3ef81201008 100644
--- a/src/backend/commands/copyfrom.c
+++ b/src/backend/commands/copyfrom.c
@@ -1721,6 +1721,12 @@ BeginCopyFrom(ParseState *pstate,
cstate->cur_attval = NULL;
cstate->relname_only = false;
+ /* Initialize SIMD variables */
+ cstate->simd_continue = false;
+ cstate->simd_initialized = false;
+ cstate->simd_current_sleep_cycle = 0;
+ cstate->simd_last_sleep_cycle = 0;
+
/*
* Allocate buffers for the input pipeline.
*
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index 5868a7fa11f..3694b814a85 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -71,7 +71,9 @@
#include "mb/pg_wchar.h"
#include "miscadmin.h"
#include "pgstat.h"
+#include "port/pg_bitutils.h"
#include "port/pg_bswap.h"
+#include "port/simd.h"
#include "utils/builtins.h"
#include "utils/rel.h"
@@ -135,18 +137,22 @@ if (1) \
} \
} else ((void) 0)
+#define LINE_BUF_FLUSH_AFTER 4096
+
/* NOTE: there's a copy of this in copyto.c */
static const char BinarySignature[11] = "PGCOPY\n\377\r\n\0";
/* non-export function prototypes */
static bool CopyReadLine(CopyFromState cstate, bool is_csv);
-static bool CopyReadLineText(CopyFromState cstate, bool is_csv);
static int CopyReadAttributesText(CopyFromState cstate);
static int CopyReadAttributesCSV(CopyFromState cstate);
static Datum CopyReadBinaryAttribute(CopyFromState cstate, FmgrInfo *flinfo,
Oid typioparam, int32 typmod,
bool *isnull);
+static pg_attribute_always_inline bool CopyReadLineText(CopyFromState cstate,
+ bool is_csv,
+ bool simd_continue);
static pg_attribute_always_inline bool CopyFromTextLikeOneRow(CopyFromState cstate,
ExprContext *econtext,
Datum *values,
@@ -1171,8 +1177,23 @@ CopyReadLine(CopyFromState cstate, bool is_csv)
resetStringInfo(&cstate->line_buf);
cstate->line_buf_valid = false;
- /* Parse data and transfer into line_buf */
- result = CopyReadLineText(cstate, is_csv);
+ /* If that is the first time we do read, initalize the SIMD */
+ if (unlikely(!cstate->simd_initialized))
+ {
+ cstate->simd_initialized = true;
+ cstate->simd_continue = true;
+ cstate->simd_current_sleep_cycle = 0;
+ cstate->simd_last_sleep_cycle = 0;
+ }
+
+ /*
+ * Parse data and transfer into line_buf. To get benefit from inlining,
+ * call CopyReadLineText() with the constant boolean variables.
+ */
+ if (cstate->simd_continue)
+ result = CopyReadLineText(cstate, is_csv, true);
+ else
+ result = CopyReadLineText(cstate, is_csv, false);
if (result)
{
@@ -1239,8 +1260,8 @@ CopyReadLine(CopyFromState cstate, bool is_csv)
/*
* CopyReadLineText - inner loop of CopyReadLine for text mode
*/
-static bool
-CopyReadLineText(CopyFromState cstate, bool is_csv)
+static pg_attribute_always_inline bool
+CopyReadLineText(CopyFromState cstate, bool is_csv, bool simd_continue)
{
char *copy_input_buf;
int input_buf_ptr;
@@ -1255,6 +1276,19 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
char quotec = '\0';
char escapec = '\0';
+#ifndef USE_NO_SIMD
+#define SIMD_SLEEP_MAX 1024
+#define SIMD_ADVANCE_AT_LEAST 5
+ Vector8 nl = vector8_broadcast('\n');
+ Vector8 cr = vector8_broadcast('\r');
+ Vector8 bs = vector8_broadcast('\\');
+ Vector8 quote = vector8_broadcast(0);
+ Vector8 escape = vector8_broadcast(0);
+
+ uint64 simd_total_cycle = 0;
+ uint64 simd_total_advance = 0;
+#endif
+
if (is_csv)
{
quotec = cstate->opts.quote[0];
@@ -1262,6 +1296,12 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
/* ignore special escape processing if it's the same as quotec */
if (quotec == escapec)
escapec = '\0';
+
+#ifndef USE_NO_SIMD
+ quote = vector8_broadcast(quotec);
+ if (quotec != escapec)
+ escape = vector8_broadcast(escapec);
+#endif
}
/*
@@ -1328,6 +1368,74 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
need_data = false;
}
+#ifndef USE_NO_SIMD
+
+ /*
+ * Use SIMD instructions to efficiently scan the input buffer for
+ * special characters (e.g., newline, carriage return, quote, and
+ * escape). This is faster than byte-by-byte iteration, especially on
+ * large buffers.
+ *
+ * We do not apply the SIMD fast path in either of the following
+ * cases: - When the previously processed character was an escape
+ * character (last_was_esc), since the next byte must be examined
+ * sequentially. - The remaining buffer is smaller than one vector
+ * width (sizeof(Vector8)); SIMD operates on fixed-size chunks.
+ */
+ if (simd_continue && !last_was_esc && copy_buf_len - input_buf_ptr >= sizeof(Vector8))
+ {
+ Vector8 chunk;
+ Vector8 match = vector8_broadcast(0);
+ uint32 mask;
+
+ simd_total_cycle++;
+
+ /* Load a chunk of data into a vector register */
+ vector8_load(&chunk, (const uint8 *) ©_input_buf[input_buf_ptr]);
+
+ /* \n and \r are not special inside quotes */
+ if (!in_quote)
+ match = vector8_or(vector8_eq(chunk, nl), vector8_eq(chunk, cr));
+
+ if (is_csv)
+ {
+ match = vector8_or(match, vector8_eq(chunk, quote));
+ if (escapec != '\0')
+ match = vector8_or(match, vector8_eq(chunk, escape));
+ }
+ else
+ match = vector8_or(match, vector8_eq(chunk, bs));
+
+ /* Check if we found any special characters */
+ mask = vector8_highbit_mask(match);
+ if (mask != 0)
+ {
+ /*
+ * Found a special character. Advance up to that point and let
+ * the scalar code handle it.
+ */
+ int advance = pg_rightmost_one_pos32(mask);
+
+ input_buf_ptr += advance;
+ simd_total_advance += advance;
+ }
+ else
+ {
+ /* No special characters found, so skip the entire chunk */
+ input_buf_ptr += sizeof(Vector8);
+ simd_total_advance += sizeof(Vector8);
+
+ // /* Periodically flush to line_buf to avoid L1D cache misses */
+ if (input_buf_ptr - cstate->input_buf_index >= LINE_BUF_FLUSH_AFTER)
+ {
+ REFILL_LINEBUF;
+ }
+
+ continue;
+ }
+ }
+#endif
+
/* OK to fetch a character */
prev_raw_ptr = input_buf_ptr;
c = copy_input_buf[input_buf_ptr++];
@@ -1530,6 +1638,50 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
}
} /* end of outer loop */
+#ifndef USE_NO_SIMD
+
+ /* SIMD was enabled */
+ if (simd_continue)
+ {
+ /* SIMD is worth */
+ if (simd_total_cycle && simd_total_advance / simd_total_cycle >= SIMD_ADVANCE_AT_LEAST)
+ {
+ Assert(cstate->simd_current_sleep_cycle == 0);
+ cstate->simd_last_sleep_cycle >>= 1;
+ }
+ /* SIMD was enabled but it isn't worth */
+ else
+ {
+ uint16 simd_last_sleep_cycle = cstate->simd_last_sleep_cycle;
+
+ cstate->simd_continue = false;
+
+ if (simd_last_sleep_cycle == 0)
+ simd_last_sleep_cycle = 1;
+ else if (simd_last_sleep_cycle >= SIMD_SLEEP_MAX / 2)
+ simd_last_sleep_cycle = SIMD_SLEEP_MAX;
+ else
+ simd_last_sleep_cycle <<= 1;
+ cstate->simd_current_sleep_cycle = simd_last_sleep_cycle;
+ cstate->simd_last_sleep_cycle = simd_last_sleep_cycle;
+ }
+ }
+ /* SIMD was disabled */
+ else
+ {
+ /*
+ * We should come here with decrementing
+ * cstate->simd_current_sleep_cycle from a positive number.
+ */
+ Assert(cstate->simd_current_sleep_cycle != 0);
+ cstate->simd_current_sleep_cycle--;
+
+ if (cstate->simd_current_sleep_cycle == 0)
+ cstate->simd_continue = true;
+ }
+
+#endif
+
/*
* Transfer any still-uncopied data to line_buf.
*/
diff --git a/src/include/commands/copyfrom_internal.h b/src/include/commands/copyfrom_internal.h
index 822ef33cf69..0d72e03c999 100644
--- a/src/include/commands/copyfrom_internal.h
+++ b/src/include/commands/copyfrom_internal.h
@@ -89,6 +89,13 @@ typedef struct CopyFromStateData
const char *cur_attval; /* current att value for error messages */
bool relname_only; /* don't output line number, att, etc. */
+ /* SIMD variables */
+ bool simd_continue;
+ bool simd_initialized;
+ uint16 simd_last_sleep_cycle;
+ uint16 simd_current_sleep_cycle;
+
+
/*
* Working state
*/
--
2.34.1
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 22:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-10 11:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-21 20:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Neil Conway <neil.conway@gmail.com>
2026-01-31 16:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
@ 2026-02-02 14:17 ` Manni Wood <manni.wood@enterprisedb.com>
1 sibling, 0 replies; 178+ messages in thread
From: Manni Wood @ 2026-02-02 14:17 UTC (permalink / raw)
To: KAZAR Ayoub <ma_kazar@esi.dz>; +Cc: Neil Conway <neil.conway@gmail.com>; Nazir Bilal Yavuz <byavuz81@gmail.com>; Nathan Bossart <nathandbossart@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
On Sat, Jan 31, 2026 at 10:21 AM KAZAR Ayoub <ma_kazar@esi.dz> wrote:
> Hello,
>
> On Wed, Jan 21, 2026 at 9:50 PM Neil Conway <neil.conway@gmail.com> wrote:
>
>> A few suggestions:
>>
>> * I'm curious if we'll see better performance on large inputs if we flush
>> to `line_buf` periodically (e.g., at least every few thousand bytes or so).
>> Otherwise we might see poor data cache behavior if large inputs with no
>> control characters get evicted before we've copied them over. See the
>> approach taken in escape_json_with_len() in utils/adt/json.c
>>
>> So i gave this a try, attached is the small patch that has v3 + the
> suggestion added, here are the results with different threshold for
> line_buf refill:
>
> Execution time compared to master:
> Workload v3 v3.1 (2k) v3.1 (4k) v3.1 (8k) v3.1 (16k) v3.1 (20k) v3.1 (28k)
> text/none -16.5% -17.4% -14.3% -12.6% -13.6% -10.5% -16.3%
> text/esc +5.6% +11.1% +3.1% +7.6% +3.0% +4.9% +4.2%
> csv/none -31.0% -29.9% -26.7% -30.1% -27.9% -30.2% -29.6%
> csv/quote +0.2% -0.6% -0.4% -1.0% +0.1% +2.5% -1.0%
>
> L1d cache miss rates:
> Workload Master v3 v3.1 (2k) v3.1 (4k) v3.1 (8k) v3.1 (16k) v3.1 (20k) v3.1
> (28k)
> text/none 0.20% 0.23% 0.21% 0.22% 0.21% 0.21% 0.21% 0.22%
> text/esc 0.21% 0.22% 0.22% 0.22% 0.22% 0.21% 0.22% 0.22%
> csv/none 0.17% 0.22% 0.21% 0.22% 0.21% 0.21% 0.22% 0.22%
> csv/quote 0.18% 0.22% 0.19% 0.20% 0.20% 0.19% 0.20% 0.20%
> On my laptop I have 32KB L1 cache per core.
> Results are super close, it is hard to see in the cache misses numbers but
> execution times are saying other things, doing the periodic filling of
> line_buf seems good to do.
> If Manni can rerun the benchmarks on these too, it would be nice to
> confirm this.
>
>
> Regards,
> Ayoub
>
Hello, All!
Ayoub, I will try to benchmark v3.1 this week on my standalone x86 and arm
PCs. Sadly, other work has been taking priority these last couple weeks,
but I will carve out some time.
Neil, thanks so much for looking at this patch!
-Manni
--
-- Manni Wood EDB: https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 22:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-10 11:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-21 20:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Neil Conway <neil.conway@gmail.com>
2026-01-31 16:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
@ 2026-02-06 13:51 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 18:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
1 sibling, 1 reply; 178+ messages in thread
From: Nazir Bilal Yavuz @ 2026-02-06 13:51 UTC (permalink / raw)
To: KAZAR Ayoub <ma_kazar@esi.dz>; +Cc: Neil Conway <neil.conway@gmail.com>; Manni Wood <manni.wood@enterprisedb.com>; Nathan Bossart <nathandbossart@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
Hi,
On Sat, 31 Jan 2026 at 19:21, KAZAR Ayoub <ma_kazar@esi.dz> wrote:
>
> On Wed, Jan 21, 2026 at 9:50 PM Neil Conway <neil.conway@gmail.com> wrote:
>>
>> * I'm curious if we'll see better performance on large inputs if we flush to `line_buf` periodically (e.g., at least every few thousand bytes or so). Otherwise we might see poor data cache behavior if large inputs with no control characters get evicted before we've copied them over. See the approach taken in escape_json_with_len() in utils/adt/json.c
>>
> So i gave this a try, attached is the small patch that has v3 + the suggestion added, here are the results with different threshold for line_buf refill:
>
> Execution time compared to master:
> Workloadv3v3.1 (2k)v3.1 (4k)v3.1 (8k)v3.1 (16k)v3.1 (20k)v3.1 (28k)
> text/none-16.5%-17.4%-14.3%-12.6%-13.6%-10.5%-16.3%
> text/esc+5.6%+11.1%+3.1%+7.6%+3.0%+4.9%+4.2%
> csv/none-31.0%-29.9%-26.7%-30.1%-27.9%-30.2%-29.6%
> csv/quote+0.2%-0.6%-0.4%-1.0%+0.1%+2.5%-1.0%
>
> L1d cache miss rates:
> WorkloadMasterv3v3.1 (2k)v3.1 (4k)v3.1 (8k)v3.1 (16k)v3.1 (20k)v3.1 (28k)
> text/none0.20%0.23%0.21%0.22%0.21%0.21%0.21%0.22%
> text/esc0.21%0.22%0.22%0.22%0.22%0.21%0.22%0.22%
> csv/none0.17%0.22%0.21%0.22%0.21%0.21%0.22%0.22%
> csv/quote0.18%0.22%0.19%0.20%0.20%0.19%0.20%0.20%
>
> On my laptop I have 32KB L1 cache per core.
> Results are super close, it is hard to see in the cache misses numbers but execution times are saying other things, doing the periodic filling of line_buf seems good to do.
> If Manni can rerun the benchmarks on these too, it would be nice to confirm this.
I looked at this change and had a couple of points.
We already have REFILL_LINEBUF at the start of the for loop in the
CopyReadLineText() function (let’s call this refill #1). This refills
when the input_buf_ptr >= copy_buf_len check is true. On my end,
copy_buf_len stays at 8191 until the end of the input, and then it
becomes the remaining amount. So when I set LINE_BUF_FLUSH_AFTER to
8192, the REFILL_LINEBUF you added shouldn’t be called; instead,
refill #1 should be triggered.
I verified this manually by adding some logging, and the results seem
to confirm this behavior. Based on that, there shouldn’t be a
performance difference when LINE_BUF_FLUSH_AFTER >= 8k.
Could you please take a look and confirm whether you see the same behavior?
Also, I noticed that json.c uses ESCAPE_JSON_FLUSH_AFTER set to 512,
so it might be worth trying smaller values here as well.
--
Regards,
Nazir Bilal Yavuz
Microsoft
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 22:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-10 11:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-21 20:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Neil Conway <neil.conway@gmail.com>
2026-01-31 16:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 13:51 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
@ 2026-02-06 18:11 ` KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 21:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
0 siblings, 1 reply; 178+ messages in thread
From: KAZAR Ayoub @ 2026-02-06 18:11 UTC (permalink / raw)
To: Nazir Bilal Yavuz <byavuz81@gmail.com>; +Cc: Neil Conway <neil.conway@gmail.com>; Manni Wood <manni.wood@enterprisedb.com>; Nathan Bossart <nathandbossart@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
Hello,
On Fri, Feb 6, 2026 at 2:51 PM Nazir Bilal Yavuz <byavuz81@gmail.com> wrote:
> Hi,
>
> On Sat, 31 Jan 2026 at 19:21, KAZAR Ayoub <ma_kazar@esi.dz> wrote:
> >
> > On Wed, Jan 21, 2026 at 9:50 PM Neil Conway <neil.conway@gmail.com>
> wrote:
> >>
> >> * I'm curious if we'll see better performance on large inputs if we
> flush to `line_buf` periodically (e.g., at least every few thousand bytes
> or so). Otherwise we might see poor data cache behavior if large inputs
> with no control characters get evicted before we've copied them over. See
> the approach taken in escape_json_with_len() in utils/adt/json.c
> >>
> > So i gave this a try, attached is the small patch that has v3 + the
> suggestion added, here are the results with different threshold for
> line_buf refill:
> >
> > Execution time compared to master:
> > Workloadv3v3.1 (2k)v3.1 (4k)v3.1 (8k)v3.1 (16k)v3.1 (20k)v3.1 (28k)
> > text/none-16.5%-17.4%-14.3%-12.6%-13.6%-10.5%-16.3%
> > text/esc+5.6%+11.1%+3.1%+7.6%+3.0%+4.9%+4.2%
> > csv/none-31.0%-29.9%-26.7%-30.1%-27.9%-30.2%-29.6%
> > csv/quote+0.2%-0.6%-0.4%-1.0%+0.1%+2.5%-1.0%
> >
> > L1d cache miss rates:
> > WorkloadMasterv3v3.1 (2k)v3.1 (4k)v3.1 (8k)v3.1 (16k)v3.1 (20k)v3.1 (28k)
> > text/none0.20%0.23%0.21%0.22%0.21%0.21%0.21%0.22%
> > text/esc0.21%0.22%0.22%0.22%0.22%0.21%0.22%0.22%
> > csv/none0.17%0.22%0.21%0.22%0.21%0.21%0.22%0.22%
> > csv/quote0.18%0.22%0.19%0.20%0.20%0.19%0.20%0.20%
> >
> > On my laptop I have 32KB L1 cache per core.
> > Results are super close, it is hard to see in the cache misses numbers
> but execution times are saying other things, doing the periodic filling of
> line_buf seems good to do.
> > If Manni can rerun the benchmarks on these too, it would be nice to
> confirm this.
>
> I looked at this change and had a couple of points.
>
> We already have REFILL_LINEBUF at the start of the for loop in the
> CopyReadLineText() function (let’s call this refill #1). This refills
> when the input_buf_ptr >= copy_buf_len check is true. On my end,
> copy_buf_len stays at 8191 until the end of the input, and then it
> becomes the remaining amount. So when I set LINE_BUF_FLUSH_AFTER to
> 8192, the REFILL_LINEBUF you added shouldn’t be called; instead,
> refill #1 should be triggered.
>
> I verified this manually by adding some logging, and the results seem
> to confirm this behavior. Based on that, there shouldn’t be a
> performance difference when LINE_BUF_FLUSH_AFTER >= 8k.
>
> Could you please take a look and confirm whether you see the same behavior?
>
So just to make sure i understand this correctly, line_buf holds processed
bytes of ONE line, so for the periodic flush that i did in:
input_buf_ptr - cstate->input_buf_index >= LINE_BUF_FLUSH_AFTER
if a line in the file is smaller than LINE_BUF_FLUSH_AFTER, the #2
REFILL_LINEBUF is never reached in a CopyReadLine entrance, as line_buf is
cleared after a line:
CopyReadLine(CopyFromState cstate, bool is_csv)
{
bool result;
resetStringInfo(&cstate->line_buf);
....
}
So my previous benchmarks (ones that have LINE_BUF_FLUSH_AFTER > 4096) are
wrong since I was working with lines of 4096 bytes.
If: Line length < LINE_BUF_FLUSH_AFTER < INPUT_BUF_SIZE : This neither hits
#1 REFILL_LINEBUF nor #2 PERIODIC REFILL_LINEBUF, this only reaches the
#3 REFILL_LINEBUF before CopyReadLineText returns.
So flushing here is just mitigating against l1d cache misses for long lines
(lines that occupy for example > 70% input_buf size, maybe ?)
Does this make sense for your case too with 8k ?
So i propose removing REFILL_LINEBUF in #1, as it doesn't make sense
anymore since PERIODIC REFILL_LINEBUF already does the job for smaller
sizes than INPUT_BUF_SIZE (in accordance to most l1d cache sizes).
So basically it becomes
If (line length < LINE_BUF_FLUSH_AFTER) then flush at very end
if (line length > LINE_BUF_FLUSH_AFTER) flush (line length
/ LINE_BUF_FLUSH_AFTER) + leftover times
> Also, I noticed that json.c uses ESCAPE_JSON_FLUSH_AFTER set to 512,
> so it might be worth trying smaller values here as well.
If I'm correct about the usage of LINE_BUF_FLUSH_AFTER above, I think
smaller values would imply too many memory loads that are unnecessary, as
for 512 we aren't battling against l1d cache misses anymore, though I'll
try it in the next benchmark.
If this sounds right, I'll be re-benchmarking for multiple row sizes with
different LINE_BUF_FLUSH_AFTER sizes.
Regards,
Ayoub
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 22:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-10 11:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-21 20:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Neil Conway <neil.conway@gmail.com>
2026-01-31 16:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 13:51 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 18:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
@ 2026-02-06 21:29 ` Nathan Bossart <nathandbossart@gmail.com>
2026-02-06 22:19 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
0 siblings, 1 reply; 178+ messages in thread
From: Nathan Bossart @ 2026-02-06 21:29 UTC (permalink / raw)
To: KAZAR Ayoub <ma_kazar@esi.dz>; +Cc: Nazir Bilal Yavuz <byavuz81@gmail.com>; Neil Conway <neil.conway@gmail.com>; Manni Wood <manni.wood@enterprisedb.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
Sorry for disappearing from this thread for a while.
It looks like a lot of energy has been put into benchmarking and refining
the heuristic for deciding when to use the SIMD path so that we avoid large
regressions when there are special characters. I think this is all
valuable work, but I'm a bit concerned that we are putting the cart before
the horse. IMHO it would be better to first get the SIMD code committed
with the absolute simplest heuristic we can think of (e.g., as soon as we
see a special character, switch to the scalar path for the remainder of
COPY FROM). My hope is that would be far easier to reason about from a
performance angle. If we immediately fall back to the existing code path,
we don't need to worry about how many special characters there are and
whether they are sparse or clustered or whatever. We just need to measure
the overhead of the new branches and ensure they don't produce meaningful
regressions. Assuming that all looks good, we can then focus on the SIMD
code itself and make sure that is correct and optimal. And once we get
that portion committed, we could then consider more sophisticated
heuristics.
FWIW I'm hoping to get something in this area committed for v19, and IMO
now is a good time to start thinking about how to get things over the
finish line. Thanks for working on it.
--
nathan
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 22:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-10 11:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-21 20:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Neil Conway <neil.conway@gmail.com>
2026-01-31 16:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 13:51 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 18:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 21:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
@ 2026-02-06 22:19 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 22:36 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 22:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
0 siblings, 2 replies; 178+ messages in thread
From: Nazir Bilal Yavuz @ 2026-02-06 22:19 UTC (permalink / raw)
To: Nathan Bossart <nathandbossart@gmail.com>; +Cc: KAZAR Ayoub <ma_kazar@esi.dz>; Neil Conway <neil.conway@gmail.com>; Manni Wood <manni.wood@enterprisedb.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
Hi,
Thank you for sharing your thoughts!
On Sat, 7 Feb 2026 at 00:29, Nathan Bossart <nathandbossart@gmail.com> wrote:
>
> It looks like a lot of energy has been put into benchmarking and refining
> the heuristic for deciding when to use the SIMD path so that we avoid large
> regressions when there are special characters. I think this is all
> valuable work, but I'm a bit concerned that we are putting the cart before
> the horse. IMHO it would be better to first get the SIMD code committed
> with the absolute simplest heuristic we can think of (e.g., as soon as we
> see a special character, switch to the scalar path for the remainder of
> COPY FROM). My hope is that would be far easier to reason about from a
> performance angle. If we immediately fall back to the existing code path,
> we don't need to worry about how many special characters there are and
> whether they are sparse or clustered or whatever. We just need to measure
> the overhead of the new branches and ensure they don't produce meaningful
> regressions. Assuming that all looks good, we can then focus on the SIMD
> code itself and make sure that is correct and optimal. And once we get
> that portion committed, we could then consider more sophisticated
> heuristics.
I have three possible approaches in my mind, they are actually similar
to each other.
1- After encountering a special character, disable SIMD for the rest
of the current line and also for the rest of the data.
2- It is a mixed version of the current heuristic and #1. After
encountering a special character, skip SIMD for the current line (let'
say line 1) and for the next line (line 2). Then try running SIMD for
the next line (line 3), if there is no special character continue to
run SIMD but if there is a special character then skip running SIMD
for two lines this time. And it goes like that, everytime special
character is encountered in the SIMD run, skipped SIMD lines are
doubled.
3- This version is a bit different from #2. Instead of calculating the
number of lines to skip dynamically, skip the constant N number of
lines and then try to run SIMD again after these lines. N could be
something like 100, 1000, or 10000 etc.. Actually, you and Andrew
suggested this approach before [1].
I think what you suggested is closer to #1 or #3. I just wanted to
hear your opinions, and whether you think any of these approaches are
good to implement / work on.
[1] https://postgr.es/m/aR4wDwNdLc5TmcQq%40nathan
--
Regards,
Nazir Bilal Yavuz
Microsoft
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 22:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-10 11:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-21 20:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Neil Conway <neil.conway@gmail.com>
2026-01-31 16:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 13:51 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 18:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 21:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-06 22:19 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
@ 2026-02-06 22:36 ` KAZAR Ayoub <ma_kazar@esi.dz>
1 sibling, 0 replies; 178+ messages in thread
From: KAZAR Ayoub @ 2026-02-06 22:36 UTC (permalink / raw)
To: Nazir Bilal Yavuz <byavuz81@gmail.com>; +Cc: Nathan Bossart <nathandbossart@gmail.com>; Neil Conway <neil.conway@gmail.com>; Manni Wood <manni.wood@enterprisedb.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
Hello,
On Fri, Feb 6, 2026 at 11:19 PM Nazir Bilal Yavuz <byavuz81@gmail.com>
wrote:
> Hi,
>
> Thank you for sharing your thoughts!
>
> On Sat, 7 Feb 2026 at 00:29, Nathan Bossart <nathandbossart@gmail.com>
> wrote:
> >
> > It looks like a lot of energy has been put into benchmarking and refining
> > the heuristic for deciding when to use the SIMD path so that we avoid
> large
> > regressions when there are special characters. I think this is all
> > valuable work, but I'm a bit concerned that we are putting the cart
> before
> > the horse. IMHO it would be better to first get the SIMD code committed
> > with the absolute simplest heuristic we can think of (e.g., as soon as we
> > see a special character, switch to the scalar path for the remainder of
> > COPY FROM). My hope is that would be far easier to reason about from a
> > performance angle. If we immediately fall back to the existing code
> path,
> > we don't need to worry about how many special characters there are and
> > whether they are sparse or clustered or whatever. We just need to
> measure
> > the overhead of the new branches and ensure they don't produce meaningful
> > regressions. Assuming that all looks good, we can then focus on the SIMD
> > code itself and make sure that is correct and optimal. And once we get
> > that portion committed, we could then consider more sophisticated
> > heuristics.
>
I also agree on this, especially for the line_buf refilling idea, it needs
a bit more time to find the good value of threshold than work for
heuristic.
>
> I have three possible approaches in my mind, they are actually similar
> to each other.
>
> 1- After encountering a special character, disable SIMD for the rest
> of the current line and also for the rest of the data.
>
> 2- It is a mixed version of the current heuristic and #1. After
> encountering a special character, skip SIMD for the current line (let'
> say line 1) and for the next line (line 2). Then try running SIMD for
> the next line (line 3), if there is no special character continue to
> run SIMD but if there is a special character then skip running SIMD
> for two lines this time. And it goes like that, everytime special
> character is encountered in the SIMD run, skipped SIMD lines are
> doubled.
>
> 3- This version is a bit different from #2. Instead of calculating the
> number of lines to skip dynamically, skip the constant N number of
> lines and then try to run SIMD again after these lines. N could be
> something like 100, 1000, or 10000 etc.. Actually, you and Andrew
> suggested this approach before [1].
>
> I think what you suggested is closer to #1 or #3. I just wanted to
> hear your opinions, and whether you think any of these approaches are
> good to implement / work on.
>
For v19, #1 seems like a "wasted potential", #3 sounds more relaxed than
v4.2 so this has good potential, i can fully benchmark it against v3 as
soon as you send a patch for it.
Regards,
Ayoub
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 22:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-10 11:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-21 20:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Neil Conway <neil.conway@gmail.com>
2026-01-31 16:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 13:51 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 18:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 21:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-06 22:19 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
@ 2026-02-06 22:47 ` Nathan Bossart <nathandbossart@gmail.com>
2026-02-11 13:27 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
1 sibling, 1 reply; 178+ messages in thread
From: Nathan Bossart @ 2026-02-06 22:47 UTC (permalink / raw)
To: Nazir Bilal Yavuz <byavuz81@gmail.com>; +Cc: KAZAR Ayoub <ma_kazar@esi.dz>; Neil Conway <neil.conway@gmail.com>; Manni Wood <manni.wood@enterprisedb.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
On Sat, Feb 07, 2026 at 01:19:16AM +0300, Nazir Bilal Yavuz wrote:
> I have three possible approaches in my mind, they are actually similar
> to each other.
>
> 1- After encountering a special character, disable SIMD for the rest
> of the current line and also for the rest of the data.
>
> 2- It is a mixed version of the current heuristic and #1. After
> encountering a special character, skip SIMD for the current line (let'
> say line 1) and for the next line (line 2). Then try running SIMD for
> the next line (line 3), if there is no special character continue to
> run SIMD but if there is a special character then skip running SIMD
> for two lines this time. And it goes like that, everytime special
> character is encountered in the SIMD run, skipped SIMD lines are
> doubled.
>
> 3- This version is a bit different from #2. Instead of calculating the
> number of lines to skip dynamically, skip the constant N number of
> lines and then try to run SIMD again after these lines. N could be
> something like 100, 1000, or 10000 etc.. Actually, you and Andrew
> suggested this approach before [1].
>
> I think what you suggested is closer to #1 or #3. I just wanted to
> hear your opinions, and whether you think any of these approaches are
> good to implement / work on.
Yeah, I think either (1) or (3) would be a good starting point. (1) is
basically just (3) with N set to infinity, anyway. I imagine there's some
value less than infinity that is acceptable, but if I had to pick an
approach right now, I'd probably go with (1) to essentially remove the
heuristic from the discussion until we're ready to focus on it.
--
nathan
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 22:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-10 11:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-21 20:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Neil Conway <neil.conway@gmail.com>
2026-01-31 16:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 13:51 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 18:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 21:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-06 22:19 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 22:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
@ 2026-02-11 13:27 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-11 22:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
0 siblings, 1 reply; 178+ messages in thread
From: Nazir Bilal Yavuz @ 2026-02-11 13:27 UTC (permalink / raw)
To: Nathan Bossart <nathandbossart@gmail.com>; +Cc: KAZAR Ayoub <ma_kazar@esi.dz>; Neil Conway <neil.conway@gmail.com>; Manni Wood <manni.wood@enterprisedb.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
Hi,
On Sat, 7 Feb 2026 at 01:47, Nathan Bossart <nathandbossart@gmail.com> wrote:
>
> On Sat, Feb 07, 2026 at 01:19:16AM +0300, Nazir Bilal Yavuz wrote:
> > I have three possible approaches in my mind, they are actually similar
> > to each other.
> >
> > 1- After encountering a special character, disable SIMD for the rest
> > of the current line and also for the rest of the data.
> >
> > 2- It is a mixed version of the current heuristic and #1. After
> > encountering a special character, skip SIMD for the current line (let'
> > say line 1) and for the next line (line 2). Then try running SIMD for
> > the next line (line 3), if there is no special character continue to
> > run SIMD but if there is a special character then skip running SIMD
> > for two lines this time. And it goes like that, everytime special
> > character is encountered in the SIMD run, skipped SIMD lines are
> > doubled.
> >
> > 3- This version is a bit different from #2. Instead of calculating the
> > number of lines to skip dynamically, skip the constant N number of
> > lines and then try to run SIMD again after these lines. N could be
> > something like 100, 1000, or 10000 etc.. Actually, you and Andrew
> > suggested this approach before [1].
> >
> > I think what you suggested is closer to #1 or #3. I just wanted to
> > hear your opinions, and whether you think any of these approaches are
> > good to implement / work on.
>
> Yeah, I think either (1) or (3) would be a good starting point. (1) is
> basically just (3) with N set to infinity, anyway. I imagine there's some
> value less than infinity that is acceptable, but if I had to pick an
> approach right now, I'd probably go with (1) to essentially remove the
> heuristic from the discussion until we're ready to focus on it.
I am sharing a v6 which implements (1). My benchmark results show
almost no difference for the special-character cases and a nice
improvement for the no-special-character cases.
Timing results after running Manni's v1.2.1 benchmark:
+---------+---------------+----------------+--------------+----------------+
| | text | no sp. | text | 1/3 sp. | csv | no sp. | csv | 1/3 sp. |
+---------+---------------+----------------+--------------+----------------+
| master | 104437 | 118711 | 121173 | 151589 |
+---------+---------------+----------------+--------------+----------------+
| patched | 90062 -%13.7 | 119070 +%0.003 | 88964 -%26.5 | 153710 +%0.013 |
+---------+---------------+----------------+--------------+----------------+
In case the table does not render well in your email client, here is a
short summary:
- Text, no special characters: 13.7% faster
- Text, 1/3 special characters: %0.003 slower, no meaningful change
- CSV, no special characters: 26.5% faster
- CSV, 1/3 special characters: %0.013 slower, no meaningful change
--
Regards,
Nazir Bilal Yavuz
Microsoft
Attachments:
[text/x-patch] v6-0001-Speed-up-COPY-FROM-text-CSV-parsing-using-SIMD.patch (8.0K, ../../CAN55FZ1=O6TjeZM2CUT7T2tu66uJT+w3G9FiRXVs+gt_ousFxQ@mail.gmail.com/2-v6-0001-Speed-up-COPY-FROM-text-CSV-parsing-using-SIMD.patch)
download | inline diff:
From 494f86e2cd01c9d55e90f7683e151828d127b8e4 Mon Sep 17 00:00:00 2001
From: Nazir Bilal Yavuz <byavuz81@gmail.com>
Date: Wed, 11 Feb 2026 14:49:21 +0300
Subject: [PATCH v6] Speed up COPY FROM text/CSV parsing using SIMD
This patch disables SIMD when SIMD encounters a special character which
is neither EOF nor EOL.
Author: Shinya Kato <shinya11.kato@gmail.com>
Author: Nazir Bilal Yavuz <byavuz81@gmail.com>
Reviewed-by: Kazar Ayoub <ma_kazar@esi.dz>
Reviewed-by: Nathan Bossart <nathandbossart@gmail.com>
Reviewed-by: Neil Conway <neil.conway@gmail.com>
Reviewed-by: Andrew Dunstan <andrew@dunslane.net>
Reviewed-by: Manni Wood <manni.wood@enterprisedb.com>
Reviewed-by: Mark Wong <markwkm@gmail.com>
Discussion: https://postgr.es/m/CAOzEurSW8cNr6TPKsjrstnPfhf4QyQqB4tnPXGGe8N4e_v7Jig%40mail.gmail.com
---
src/backend/commands/copyfrom.c | 4 +
src/backend/commands/copyfromparse.c | 132 ++++++++++++++++++++++-
src/include/commands/copyfrom_internal.h | 4 +
3 files changed, 135 insertions(+), 5 deletions(-)
diff --git a/src/backend/commands/copyfrom.c b/src/backend/commands/copyfrom.c
index 25ee20b23db..fbf78b6698b 100644
--- a/src/backend/commands/copyfrom.c
+++ b/src/backend/commands/copyfrom.c
@@ -1721,6 +1721,10 @@ BeginCopyFrom(ParseState *pstate,
cstate->cur_attval = NULL;
cstate->relname_only = false;
+ /* Initialize SIMD variables */
+ cstate->simd_enabled = false;
+ cstate->simd_initialized = false;
+
/*
* Allocate buffers for the input pipeline.
*
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index 94d6f415a06..554b3cb9bf8 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -72,6 +72,7 @@
#include "miscadmin.h"
#include "pgstat.h"
#include "port/pg_bswap.h"
+#include "port/simd.h"
#include "utils/builtins.h"
#include "utils/rel.h"
@@ -141,12 +142,14 @@ static const char BinarySignature[11] = "PGCOPY\n\377\r\n\0";
/* non-export function prototypes */
static bool CopyReadLine(CopyFromState cstate, bool is_csv);
-static bool CopyReadLineText(CopyFromState cstate, bool is_csv);
static int CopyReadAttributesText(CopyFromState cstate);
static int CopyReadAttributesCSV(CopyFromState cstate);
static Datum CopyReadBinaryAttribute(CopyFromState cstate, FmgrInfo *flinfo,
Oid typioparam, int32 typmod,
bool *isnull);
+static pg_attribute_always_inline bool CopyReadLineText(CopyFromState cstate,
+ bool is_csv,
+ bool simd_enabled);
static pg_attribute_always_inline bool CopyFromTextLikeOneRow(CopyFromState cstate,
ExprContext *econtext,
Datum *values,
@@ -1173,8 +1176,21 @@ CopyReadLine(CopyFromState cstate, bool is_csv)
resetStringInfo(&cstate->line_buf);
cstate->line_buf_valid = false;
- /* Parse data and transfer into line_buf */
- result = CopyReadLineText(cstate, is_csv);
+ /* Initialize SIMD on the first read */
+ if (unlikely(!cstate->simd_initialized))
+ {
+ cstate->simd_initialized = true;
+ cstate->simd_enabled = true;
+ }
+
+ /*
+ * Parse data and transfer into line_buf. To benefit from inlining, call
+ * CopyReadLineText() with constant boolean arguments.
+ */
+ if (cstate->simd_enabled)
+ result = CopyReadLineText(cstate, is_csv, true);
+ else
+ result = CopyReadLineText(cstate, is_csv, false);
if (result)
{
@@ -1241,8 +1257,8 @@ CopyReadLine(CopyFromState cstate, bool is_csv)
/*
* CopyReadLineText - inner loop of CopyReadLine for text mode
*/
-static bool
-CopyReadLineText(CopyFromState cstate, bool is_csv)
+static pg_attribute_always_inline bool
+CopyReadLineText(CopyFromState cstate, bool is_csv, bool simd_enabled)
{
char *copy_input_buf;
int input_buf_ptr;
@@ -1257,6 +1273,14 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
char quotec = '\0';
char escapec = '\0';
+#ifndef USE_NO_SIMD
+ Vector8 nl = vector8_broadcast('\n');
+ Vector8 cr = vector8_broadcast('\r');
+ Vector8 bs = vector8_broadcast('\\');
+ Vector8 quote = vector8_broadcast(0);
+ Vector8 escape = vector8_broadcast(0);
+#endif
+
if (is_csv)
{
quotec = cstate->opts.quote[0];
@@ -1264,6 +1288,12 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
/* ignore special escape processing if it's the same as quotec */
if (quotec == escapec)
escapec = '\0';
+
+#ifndef USE_NO_SIMD
+ quote = vector8_broadcast(quotec);
+ if (quotec != escapec)
+ escape = vector8_broadcast(escapec);
+#endif
}
/*
@@ -1330,6 +1360,98 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
need_data = false;
}
+#ifndef USE_NO_SIMD
+
+ /*
+ * Use SIMD instructions to efficiently scan the input buffer for
+ * special characters (e.g., newline, carriage return, quote, and
+ * escape). This is faster than byte-by-byte iteration, especially on
+ * large buffers.
+ *
+ * We do not apply the SIMD fast path in either of the following
+ * cases: - When the previously processed character was an escape
+ * character (last_was_esc), since the next byte must be examined
+ * sequentially. - When the remaining buffer is smaller than one
+ * vector width (sizeof(Vector8)), since SIMD operates on fixed-size
+ * chunks.
+ *
+ * Note that, SIMD may become slower when the input contains many
+ * special characters. To avoid this regression, we disable SIMD for
+ * the rest of the input once we encounter a special character which
+ * is neither EOF nor EOL.
+ */
+ if (simd_enabled && !last_was_esc && copy_buf_len - input_buf_ptr > sizeof(Vector8))
+ {
+ Vector8 chunk;
+ Vector8 match = vector8_broadcast(0);
+ uint32 mask;
+
+ /* Load a chunk of data into a vector register */
+ vector8_load(&chunk, (const uint8 *) ©_input_buf[input_buf_ptr]);
+
+ if (is_csv)
+ {
+ /* \n and \r are not special inside quotes */
+ if (!in_quote)
+ match = vector8_or(vector8_eq(chunk, nl), vector8_eq(chunk, cr));
+
+ match = vector8_or(match, vector8_eq(chunk, quote));
+ if (escapec != '\0')
+ match = vector8_or(match, vector8_eq(chunk, escape));
+ }
+ else
+ {
+ match = vector8_or(vector8_eq(chunk, nl), vector8_eq(chunk, cr));
+ match = vector8_or(match, vector8_eq(chunk, bs));
+ }
+
+ /* Check if we found any special characters */
+ mask = vector8_highbit_mask(match);
+ if (mask != 0)
+ {
+ /*
+ * Found a special character. Advance up to that point and let
+ * the scalar code handle it.
+ */
+ int advance = pg_rightmost_one_pos32(mask);
+ char c1,
+ c2;
+ bool simd_hit_eol,
+ simd_hit_eof;
+
+ input_buf_ptr += advance;
+ c1 = copy_input_buf[input_buf_ptr];
+
+ /*
+ * Since we stopped within the chunk and ((copy_buf_len -
+ * input_buf_ptr) > sizeof(Vector8)) is true,
+ * copy_input_buf[input_buf_ptr + 1] is guaranteed to be
+ * readable.
+ */
+ c2 = copy_input_buf[input_buf_ptr + 1];
+ simd_hit_eol = (c1 == '\r' || c1 == '\n') && (!is_csv || !in_quote);
+ simd_hit_eof = c1 == '\\' && c2 == '.' && !is_csv;
+
+ /*
+ * Do not disable SIMD when we hit EOL or EOF characters. In
+ * practice, it does not matter for EOF because parsing ends
+ * there, but we keep the behavior consistent.
+ */
+ if (!(simd_hit_eof || simd_hit_eol))
+ {
+ simd_enabled = false;
+ cstate->simd_enabled = false;
+ }
+ }
+ else
+ {
+ /* No special characters found, so skip the entire chunk */
+ input_buf_ptr += sizeof(Vector8);
+ continue;
+ }
+ }
+#endif
+
/* OK to fetch a character */
prev_raw_ptr = input_buf_ptr;
c = copy_input_buf[input_buf_ptr++];
diff --git a/src/include/commands/copyfrom_internal.h b/src/include/commands/copyfrom_internal.h
index 822ef33cf69..56942a15469 100644
--- a/src/include/commands/copyfrom_internal.h
+++ b/src/include/commands/copyfrom_internal.h
@@ -89,6 +89,10 @@ typedef struct CopyFromStateData
const char *cur_attval; /* current att value for error messages */
bool relname_only; /* don't output line number, att, etc. */
+ /* SIMD variables */
+ bool simd_enabled;
+ bool simd_initialized;
+
/*
* Working state
*/
--
2.47.3
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 22:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-10 11:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-21 20:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Neil Conway <neil.conway@gmail.com>
2026-01-31 16:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 13:51 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 18:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 21:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-06 22:19 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 22:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-11 13:27 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
@ 2026-02-11 22:39 ` Nathan Bossart <nathandbossart@gmail.com>
2026-02-13 11:45 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
0 siblings, 1 reply; 178+ messages in thread
From: Nathan Bossart @ 2026-02-11 22:39 UTC (permalink / raw)
To: Nazir Bilal Yavuz <byavuz81@gmail.com>; +Cc: KAZAR Ayoub <ma_kazar@esi.dz>; Neil Conway <neil.conway@gmail.com>; Manni Wood <manni.wood@enterprisedb.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
On Wed, Feb 11, 2026 at 04:27:50PM +0300, Nazir Bilal Yavuz wrote:
> I am sharing a v6 which implements (1). My benchmark results show
> almost no difference for the special-character cases and a nice
> improvement for the no-special-character cases.
Thanks!
> + /* Initialize SIMD variables */
> + cstate->simd_enabled = false;
> + cstate->simd_initialized = false;
> + /* Initialize SIMD on the first read */
> + if (unlikely(!cstate->simd_initialized))
> + {
> + cstate->simd_initialized = true;
> + cstate->simd_enabled = true;
> + }
Why do we do this initialization in CopyReadLine() as opposed to setting
simd_enabled to true when initializing cstate in BeginCopyFrom()? If we
can initialize it in BeginCopyFrom, we could probably remove
simd_initialized.
> + if (cstate->simd_enabled)
> + result = CopyReadLineText(cstate, is_csv, true);
> + else
> + result = CopyReadLineText(cstate, is_csv, false);
I know we discussed this upthread, but I'd like to take a closer look at
this to see whether/why it makes such a big difference. It's a bit awkward
that CopyReadLineText() needs to manage both its local simd_enabled and
cstate->simd_enabled.
+ /* Load a chunk of data into a vector register */
+ vector8_load(&chunk, (const uint8 *) ©_input_buf[input_buf_ptr]);
As mentioned upthread [0], I think it's worth testing whether processing
multiple vectors worth of data in each loop iteration is worthwhile.
[0] https://postgr.es/m/aSTVOe6BIe5f1l3i%40nathan
--
nathan
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 22:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-10 11:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-21 20:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Neil Conway <neil.conway@gmail.com>
2026-01-31 16:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 13:51 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 18:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 21:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-06 22:19 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 22:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-11 13:27 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-11 22:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
@ 2026-02-13 11:45 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-13 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
0 siblings, 1 reply; 178+ messages in thread
From: Nazir Bilal Yavuz @ 2026-02-13 11:45 UTC (permalink / raw)
To: Nathan Bossart <nathandbossart@gmail.com>; +Cc: KAZAR Ayoub <ma_kazar@esi.dz>; Neil Conway <neil.conway@gmail.com>; Manni Wood <manni.wood@enterprisedb.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
Hi,
Thanks for the review!
On Thu, 12 Feb 2026 at 01:39, Nathan Bossart <nathandbossart@gmail.com> wrote:
>
> On Wed, Feb 11, 2026 at 04:27:50PM +0300, Nazir Bilal Yavuz wrote:
> > I am sharing a v6 which implements (1). My benchmark results show
> > almost no difference for the special-character cases and a nice
> > improvement for the no-special-character cases.
>
> Thanks!
>
> > + /* Initialize SIMD variables */
> > + cstate->simd_enabled = false;
> > + cstate->simd_initialized = false;
>
> > + /* Initialize SIMD on the first read */
> > + if (unlikely(!cstate->simd_initialized))
> > + {
> > + cstate->simd_initialized = true;
> > + cstate->simd_enabled = true;
> > + }
>
> Why do we do this initialization in CopyReadLine() as opposed to setting
> simd_enabled to true when initializing cstate in BeginCopyFrom()? If we
> can initialize it in BeginCopyFrom, we could probably remove
> simd_initialized.
Correct, I guess this is left over from the earlier versions.
> > + if (cstate->simd_enabled)
> > + result = CopyReadLineText(cstate, is_csv, true);
> > + else
> > + result = CopyReadLineText(cstate, is_csv, false);
>
> I know we discussed this upthread, but I'd like to take a closer look at
> this to see whether/why it makes such a big difference. It's a bit awkward
> that CopyReadLineText() needs to manage both its local simd_enabled and
> cstate->simd_enabled.
I extensively benchmarked this with the new v6 version. If I change
this to either of:
CopyReadLineText(cstate, is_csv);
or
CopyReadLineText(cstate, is_csv, cstate->simd_enabled);
then there is %5-%10 regression for the scalar path. I ran my
benchmarks with both "meson --buildtype=debugoptimized" and "meson
--buildtype=release" but the result is the same.
Also, if I change this code to:
if (cstate->simd_enabled)
{
if (is_csv)
result = CopyReadLineText(cstate, true, true);
else
result = CopyReadLineText(cstate, false, true);
}
else
{
if (is_csv)
result = CopyReadLineText(cstate, true, false);
else
result = CopyReadLineText(cstate, false, false);
}
then I see ~%5 performance improvement in scalar path compared to master.
> + /* Load a chunk of data into a vector register */
> + vector8_load(&chunk, (const uint8 *) ©_input_buf[input_buf_ptr]);
>
> As mentioned upthread [0], I think it's worth testing whether processing
> multiple vectors worth of data in each loop iteration is worthwhile.
>
> [0] https://postgr.es/m/aSTVOe6BIe5f1l3i%40nathan
There are multiple keys in CopyReadLineText() compared to
pg_lfind32(). I am not sure if I correctly used multiple vectors but I
attached what I did as 0002, could you please look at it? I didn't see
any performance benefit in my benchmarks, though.
--
Regards,
Nazir Bilal Yavuz
Microsoft
Attachments:
[text/x-patch] v7-0001-Speed-up-COPY-FROM-text-CSV-parsing-using-SIMD.patch (7.8K, ../../CAN55FZ3g6QaiC8G4GMjdJ24egvgc-HG_xpoOztxnM_wnQNn5aw@mail.gmail.com/2-v7-0001-Speed-up-COPY-FROM-text-CSV-parsing-using-SIMD.patch)
download | inline diff:
From c4b29849ad9f87f51022b947a9a0ab695dd1cde2 Mon Sep 17 00:00:00 2001
From: Nazir Bilal Yavuz <byavuz81@gmail.com>
Date: Fri, 13 Feb 2026 13:28:55 +0300
Subject: [PATCH v7 1/2] Speed up COPY FROM text/CSV parsing using SIMD
This patch disables SIMD when SIMD encounters a special character which
is neither EOF nor EOL.
Author: Shinya Kato <shinya11.kato@gmail.com>
Author: Nazir Bilal Yavuz <byavuz81@gmail.com>
Reviewed-by: Kazar Ayoub <ma_kazar@esi.dz>
Reviewed-by: Nathan Bossart <nathandbossart@gmail.com>
Reviewed-by: Neil Conway <neil.conway@gmail.com>
Reviewed-by: Andrew Dunstan <andrew@dunslane.net>
Reviewed-by: Manni Wood <manni.wood@enterprisedb.com>
Reviewed-by: Mark Wong <markwkm@gmail.com>
Discussion: https://postgr.es/m/CAOzEurSW8cNr6TPKsjrstnPfhf4QyQqB4tnPXGGe8N4e_v7Jig%40mail.gmail.com
---
src/backend/commands/copyfrom.c | 3 +
src/backend/commands/copyfromparse.c | 125 ++++++++++++++++++++++-
src/include/commands/copyfrom_internal.h | 3 +
3 files changed, 126 insertions(+), 5 deletions(-)
diff --git a/src/backend/commands/copyfrom.c b/src/backend/commands/copyfrom.c
index 25ee20b23db..40dae0bdacc 100644
--- a/src/backend/commands/copyfrom.c
+++ b/src/backend/commands/copyfrom.c
@@ -1721,6 +1721,9 @@ BeginCopyFrom(ParseState *pstate,
cstate->cur_attval = NULL;
cstate->relname_only = false;
+ /* Initialize SIMD */
+ cstate->simd_enabled = true;
+
/*
* Allocate buffers for the input pipeline.
*
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index 94d6f415a06..4a127d1af90 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -72,6 +72,7 @@
#include "miscadmin.h"
#include "pgstat.h"
#include "port/pg_bswap.h"
+#include "port/simd.h"
#include "utils/builtins.h"
#include "utils/rel.h"
@@ -141,12 +142,14 @@ static const char BinarySignature[11] = "PGCOPY\n\377\r\n\0";
/* non-export function prototypes */
static bool CopyReadLine(CopyFromState cstate, bool is_csv);
-static bool CopyReadLineText(CopyFromState cstate, bool is_csv);
static int CopyReadAttributesText(CopyFromState cstate);
static int CopyReadAttributesCSV(CopyFromState cstate);
static Datum CopyReadBinaryAttribute(CopyFromState cstate, FmgrInfo *flinfo,
Oid typioparam, int32 typmod,
bool *isnull);
+static pg_attribute_always_inline bool CopyReadLineText(CopyFromState cstate,
+ bool is_csv,
+ bool simd_enabled);
static pg_attribute_always_inline bool CopyFromTextLikeOneRow(CopyFromState cstate,
ExprContext *econtext,
Datum *values,
@@ -1173,8 +1176,14 @@ CopyReadLine(CopyFromState cstate, bool is_csv)
resetStringInfo(&cstate->line_buf);
cstate->line_buf_valid = false;
- /* Parse data and transfer into line_buf */
- result = CopyReadLineText(cstate, is_csv);
+ /*
+ * Parse data and transfer into line_buf. To benefit from inlining, call
+ * CopyReadLineText() with constant boolean arguments.
+ */
+ if (cstate->simd_enabled)
+ result = CopyReadLineText(cstate, is_csv, true);
+ else
+ result = CopyReadLineText(cstate, is_csv, false);
if (result)
{
@@ -1241,8 +1250,8 @@ CopyReadLine(CopyFromState cstate, bool is_csv)
/*
* CopyReadLineText - inner loop of CopyReadLine for text mode
*/
-static bool
-CopyReadLineText(CopyFromState cstate, bool is_csv)
+static pg_attribute_always_inline bool
+CopyReadLineText(CopyFromState cstate, bool is_csv, bool simd_enabled)
{
char *copy_input_buf;
int input_buf_ptr;
@@ -1257,6 +1266,14 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
char quotec = '\0';
char escapec = '\0';
+#ifndef USE_NO_SIMD
+ Vector8 nl = vector8_broadcast('\n');
+ Vector8 cr = vector8_broadcast('\r');
+ Vector8 bs = vector8_broadcast('\\');
+ Vector8 quote = vector8_broadcast(0);
+ Vector8 escape = vector8_broadcast(0);
+#endif
+
if (is_csv)
{
quotec = cstate->opts.quote[0];
@@ -1264,6 +1281,12 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
/* ignore special escape processing if it's the same as quotec */
if (quotec == escapec)
escapec = '\0';
+
+#ifndef USE_NO_SIMD
+ quote = vector8_broadcast(quotec);
+ if (quotec != escapec)
+ escape = vector8_broadcast(escapec);
+#endif
}
/*
@@ -1330,6 +1353,98 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
need_data = false;
}
+#ifndef USE_NO_SIMD
+
+ /*
+ * Use SIMD instructions to efficiently scan the input buffer for
+ * special characters (e.g., newline, carriage return, quote, and
+ * escape). This is faster than byte-by-byte iteration, especially on
+ * large buffers.
+ *
+ * We do not apply the SIMD fast path in either of the following
+ * cases: - When the previously processed character was an escape
+ * character (last_was_esc), since the next byte must be examined
+ * sequentially. - When the remaining buffer is smaller than one
+ * vector width (sizeof(Vector8)), since SIMD operates on fixed-size
+ * chunks.
+ *
+ * Note that, SIMD may become slower when the input contains many
+ * special characters. To avoid this regression, we disable SIMD for
+ * the rest of the input once we encounter a special character which
+ * is neither EOF nor EOL.
+ */
+ if (simd_enabled && !last_was_esc && copy_buf_len - input_buf_ptr > sizeof(Vector8))
+ {
+ Vector8 chunk;
+ Vector8 match = vector8_broadcast(0);
+ uint32 mask;
+
+ /* Load a chunk of data into a vector register */
+ vector8_load(&chunk, (const uint8 *) ©_input_buf[input_buf_ptr]);
+
+ if (is_csv)
+ {
+ /* \n and \r are not special inside quotes */
+ if (!in_quote)
+ match = vector8_or(vector8_eq(chunk, nl), vector8_eq(chunk, cr));
+
+ match = vector8_or(match, vector8_eq(chunk, quote));
+ if (escapec != '\0')
+ match = vector8_or(match, vector8_eq(chunk, escape));
+ }
+ else
+ {
+ match = vector8_or(vector8_eq(chunk, nl), vector8_eq(chunk, cr));
+ match = vector8_or(match, vector8_eq(chunk, bs));
+ }
+
+ /* Check if we found any special characters */
+ mask = vector8_highbit_mask(match);
+ if (mask != 0)
+ {
+ /*
+ * Found a special character. Advance up to that point and let
+ * the scalar code handle it.
+ */
+ int advance = pg_rightmost_one_pos32(mask);
+ char c1,
+ c2;
+ bool simd_hit_eol,
+ simd_hit_eof;
+
+ input_buf_ptr += advance;
+ c1 = copy_input_buf[input_buf_ptr];
+
+ /*
+ * Since we stopped within the chunk and ((copy_buf_len -
+ * input_buf_ptr) > sizeof(Vector8)) is true,
+ * copy_input_buf[input_buf_ptr + 1] is guaranteed to be
+ * readable.
+ */
+ c2 = copy_input_buf[input_buf_ptr + 1];
+ simd_hit_eol = (c1 == '\r' || c1 == '\n') && (!is_csv || !in_quote);
+ simd_hit_eof = c1 == '\\' && c2 == '.' && !is_csv;
+
+ /*
+ * Do not disable SIMD when we hit EOL or EOF characters. In
+ * practice, it does not matter for EOF because parsing ends
+ * there, but we keep the behavior consistent.
+ */
+ if (!(simd_hit_eof || simd_hit_eol))
+ {
+ simd_enabled = false;
+ cstate->simd_enabled = false;
+ }
+ }
+ else
+ {
+ /* No special characters found, so skip the entire chunk */
+ input_buf_ptr += sizeof(Vector8);
+ continue;
+ }
+ }
+#endif
+
/* OK to fetch a character */
prev_raw_ptr = input_buf_ptr;
c = copy_input_buf[input_buf_ptr++];
diff --git a/src/include/commands/copyfrom_internal.h b/src/include/commands/copyfrom_internal.h
index 822ef33cf69..73ce777c52b 100644
--- a/src/include/commands/copyfrom_internal.h
+++ b/src/include/commands/copyfrom_internal.h
@@ -89,6 +89,9 @@ typedef struct CopyFromStateData
const char *cur_attval; /* current att value for error messages */
bool relname_only; /* don't output line number, att, etc. */
+ /* SIMD variables */
+ bool simd_enabled;
+
/*
* Working state
*/
--
2.47.3
[text/x-patch] v7-0002-Use-4-vectors-in-CopyReadLineText-SIMD.patch (6.4K, ../../CAN55FZ3g6QaiC8G4GMjdJ24egvgc-HG_xpoOztxnM_wnQNn5aw@mail.gmail.com/3-v7-0002-Use-4-vectors-in-CopyReadLineText-SIMD.patch)
download | inline diff:
From 2de9b5bc18bfa169b3ba3507b6bdf79d277c0ad4 Mon Sep 17 00:00:00 2001
From: Nazir Bilal Yavuz <byavuz81@gmail.com>
Date: Fri, 13 Feb 2026 13:36:34 +0300
Subject: [PATCH v7 2/2] Use 4 vectors in CopyReadLineText() SIMD
---
src/backend/commands/copyfromparse.c | 116 +++++++++++++++++++++------
1 file changed, 92 insertions(+), 24 deletions(-)
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index 4a127d1af90..caadc40cc8b 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -1361,6 +1361,9 @@ CopyReadLineText(CopyFromState cstate, bool is_csv, bool simd_enabled)
* escape). This is faster than byte-by-byte iteration, especially on
* large buffers.
*
+ * For better instruction-level parallelism, we try to process four
+ * vectors at a time.
+ *
* We do not apply the SIMD fast path in either of the following
* cases: - When the previously processed character was an escape
* character (last_was_esc), since the next byte must be examined
@@ -1373,53 +1376,118 @@ CopyReadLineText(CopyFromState cstate, bool is_csv, bool simd_enabled)
* the rest of the input once we encounter a special character which
* is neither EOF nor EOL.
*/
- if (simd_enabled && !last_was_esc && copy_buf_len - input_buf_ptr > sizeof(Vector8))
+ if (simd_enabled && !last_was_esc && copy_buf_len - input_buf_ptr >= 4 * sizeof(Vector8))
{
- Vector8 chunk;
- Vector8 match = vector8_broadcast(0);
- uint32 mask;
-
- /* Load a chunk of data into a vector register */
- vector8_load(&chunk, (const uint8 *) ©_input_buf[input_buf_ptr]);
+ Vector8 chunk1,
+ chunk2,
+ chunk3,
+ chunk4;
+ Vector8 match1,
+ match2,
+ match3,
+ match4;
+ Vector8 tmp1,
+ tmp2,
+ result;
+
+ /* Load four chunks of data into vector registers */
+ vector8_load(&chunk1, (const uint8 *) ©_input_buf[input_buf_ptr]);
+ vector8_load(&chunk2, (const uint8 *) ©_input_buf[input_buf_ptr + sizeof(Vector8)]);
+ vector8_load(&chunk3, (const uint8 *) ©_input_buf[input_buf_ptr + 2 * sizeof(Vector8)]);
+ vector8_load(&chunk4, (const uint8 *) ©_input_buf[input_buf_ptr + 3 * sizeof(Vector8)]);
if (is_csv)
{
/* \n and \r are not special inside quotes */
if (!in_quote)
- match = vector8_or(vector8_eq(chunk, nl), vector8_eq(chunk, cr));
+ {
+ match1 = vector8_or(vector8_eq(chunk1, nl), vector8_eq(chunk1, cr));
+ match2 = vector8_or(vector8_eq(chunk2, nl), vector8_eq(chunk2, cr));
+ match3 = vector8_or(vector8_eq(chunk3, nl), vector8_eq(chunk3, cr));
+ match4 = vector8_or(vector8_eq(chunk4, nl), vector8_eq(chunk4, cr));
+ }
+ else
+ {
+ match1 = vector8_broadcast(0);
+ match2 = vector8_broadcast(0);
+ match3 = vector8_broadcast(0);
+ match4 = vector8_broadcast(0);
+ }
- match = vector8_or(match, vector8_eq(chunk, quote));
+ match1 = vector8_or(match1, vector8_eq(chunk1, quote));
+ match2 = vector8_or(match2, vector8_eq(chunk2, quote));
+ match3 = vector8_or(match3, vector8_eq(chunk3, quote));
+ match4 = vector8_or(match4, vector8_eq(chunk4, quote));
if (escapec != '\0')
- match = vector8_or(match, vector8_eq(chunk, escape));
+ {
+ match1 = vector8_or(match1, vector8_eq(chunk1, escape));
+ match2 = vector8_or(match2, vector8_eq(chunk2, escape));
+ match3 = vector8_or(match3, vector8_eq(chunk3, escape));
+ match4 = vector8_or(match4, vector8_eq(chunk4, escape));
+ }
}
else
{
- match = vector8_or(vector8_eq(chunk, nl), vector8_eq(chunk, cr));
- match = vector8_or(match, vector8_eq(chunk, bs));
+ match1 = vector8_or(vector8_eq(chunk1, nl), vector8_eq(chunk1, cr));
+ match2 = vector8_or(vector8_eq(chunk2, nl), vector8_eq(chunk2, cr));
+ match3 = vector8_or(vector8_eq(chunk3, nl), vector8_eq(chunk3, cr));
+ match4 = vector8_or(vector8_eq(chunk4, nl), vector8_eq(chunk4, cr));
+
+ match1 = vector8_or(match1, vector8_eq(chunk1, bs));
+ match2 = vector8_or(match2, vector8_eq(chunk2, bs));
+ match3 = vector8_or(match3, vector8_eq(chunk3, bs));
+ match4 = vector8_or(match4, vector8_eq(chunk4, bs));
}
- /* Check if we found any special characters */
- mask = vector8_highbit_mask(match);
- if (mask != 0)
+ /* Combine results to check if any chunk has special characters */
+ tmp1 = vector8_or(match1, match2);
+ tmp2 = vector8_or(match3, match4);
+ result = vector8_or(tmp1, tmp2);
+
+ if (vector8_is_highbit_set(result))
{
/*
- * Found a special character. Advance up to that point and let
- * the scalar code handle it.
+ * Found a special character somewhere in the four chunks.
+ * Identify the first chunk containing it.
*/
- int advance = pg_rightmost_one_pos32(mask);
+ uint32 mask;
+ int advance;
char c1,
c2;
bool simd_hit_eol,
simd_hit_eof;
+ mask = vector8_highbit_mask(match1);
+ if (mask == 0)
+ {
+ input_buf_ptr += sizeof(Vector8);
+ mask = vector8_highbit_mask(match2);
+ }
+ if (mask == 0)
+ {
+ input_buf_ptr += sizeof(Vector8);
+ mask = vector8_highbit_mask(match3);
+ }
+ if (mask == 0)
+ {
+ input_buf_ptr += sizeof(Vector8);
+ mask = vector8_highbit_mask(match4);
+ }
+ Assert(mask != 0);
+
+ /*
+ * Found a special character. Advance up to that point and let
+ * the scalar code handle it.
+ */
+ advance = pg_rightmost_one_pos32(mask);
input_buf_ptr += advance;
c1 = copy_input_buf[input_buf_ptr];
/*
- * Since we stopped within the chunk and ((copy_buf_len -
- * input_buf_ptr) > sizeof(Vector8)) is true,
- * copy_input_buf[input_buf_ptr + 1] is guaranteed to be
- * readable.
+ * Since we stopped within the block and ((copy_buf_len -
+ * input_buf_ptr) >= 4 * sizeof(Vector8)) was true at the
+ * start, copy_input_buf[input_buf_ptr + 1] is guaranteed to
+ * be readable.
*/
c2 = copy_input_buf[input_buf_ptr + 1];
simd_hit_eol = (c1 == '\r' || c1 == '\n') && (!is_csv || !in_quote);
@@ -1438,8 +1506,8 @@ CopyReadLineText(CopyFromState cstate, bool is_csv, bool simd_enabled)
}
else
{
- /* No special characters found, so skip the entire chunk */
- input_buf_ptr += sizeof(Vector8);
+ /* No special characters found, so skip the entire block */
+ input_buf_ptr += 4 * sizeof(Vector8);
continue;
}
}
--
2.47.3
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 22:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-10 11:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-21 20:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Neil Conway <neil.conway@gmail.com>
2026-01-31 16:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 13:51 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 18:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 21:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-06 22:19 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 22:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-11 13:27 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-11 22:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-13 11:45 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
@ 2026-02-13 23:09 ` Nathan Bossart <nathandbossart@gmail.com>
2026-02-14 03:34 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-18 13:38 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-20 10:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
0 siblings, 3 replies; 178+ messages in thread
From: Nathan Bossart @ 2026-02-13 23:09 UTC (permalink / raw)
To: Nazir Bilal Yavuz <byavuz81@gmail.com>; +Cc: KAZAR Ayoub <ma_kazar@esi.dz>; Neil Conway <neil.conway@gmail.com>; Manni Wood <manni.wood@enterprisedb.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
On Fri, Feb 13, 2026 at 02:45:30PM +0300, Nazir Bilal Yavuz wrote:
> Also, if I change this code to:
>
> if (cstate->simd_enabled)
> {
> if (is_csv)
> result = CopyReadLineText(cstate, true, true);
> else
> result = CopyReadLineText(cstate, false, true);
> }
> else
> {
> if (is_csv)
> result = CopyReadLineText(cstate, true, false);
> else
> result = CopyReadLineText(cstate, false, false);
> }
>
> then I see ~%5 performance improvement in scalar path compared to master.
Hm. What difference do you see if you just do
if (is_csv)
result = CopyReadLineText(cstate, true);
else
result = CopyReadLineText(cstate, false);
both with and without the SIMD stuff? IIUC this is allowing the compiler
to remove several branches in CopyReadLineText(), which might be a nice
improvement on its own. That being said, I'm less convinced that adding a
simd_enabled parameter to CopyReadLineText() helps, because 1) it's
involved in fewer branches and 2) we change it within the function, so the
compiler can't remove the branches, anyway. But perhaps I'm missing
something.
Some other random thoughts:
+ match = vector8_or(vector8_eq(chunk, nl), vector8_eq(chunk, cr));
+ match = vector8_or(vector8_eq(chunk, nl), vector8_eq(chunk, cr));
Since \n and \r are well below "normal" ASCII values, I wonder if we could
simplify these to something like
match = vector8_gt(... vector with all lanes set to \r + 1 ..., chunk);
+ /* Check if we found any special characters */
+ mask = vector8_highbit_mask(match);
+ if (mask != 0)
vector8_highbit_mask() is somewhat expensive on AArch64, so I wonder if
waiting until we enter the "if" block to calculate it has any benefit.
+ simd_hit_eol = (c1 == '\r' || c1 == '\n') && (!is_csv || !in_quote);
If (is_csv && in_quote), we shouldn't have picked up \r or \n in the first
place, right?
+ simd_hit_eof = c1 == '\\' && c2 == '.' && !is_csv;
+
+ /*
+ * Do not disable SIMD when we hit EOL or EOF characters. In
+ * practice, it does not matter for EOF because parsing ends
+ * there, but we keep the behavior consistent.
+ */
+ if (!(simd_hit_eof || simd_hit_eol))
I'd think that doing less unnecessary work would outweigh the benefits of
consistency for the EOF case.
--
nathan
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 22:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-10 11:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-21 20:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Neil Conway <neil.conway@gmail.com>
2026-01-31 16:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 13:51 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 18:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 21:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-06 22:19 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 22:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-11 13:27 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-11 22:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-13 11:45 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-13 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
@ 2026-02-14 03:34 ` Manni Wood <manni.wood@enterprisedb.com>
2026-02-16 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2 siblings, 1 reply; 178+ messages in thread
From: Manni Wood @ 2026-02-14 03:34 UTC (permalink / raw)
To: Nathan Bossart <nathandbossart@gmail.com>; +Cc: Nazir Bilal Yavuz <byavuz81@gmail.com>; KAZAR Ayoub <ma_kazar@esi.dz>; Neil Conway <neil.conway@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
Hello!
I ran some COPY FROM tests using master and then Nazir's v7-0001 and
v7-0002 patches applied to master.
x86 master
TXT : 29222.524250 ms
CSV : 36162.588500 ms
TXT with 1/3 escapes: 32922.649750 ms
CSV with 1/3 quotes: 47631.423750 ms
x86 v7-0001
TXT : 23247.834250 ms 20.445496% improvement
CSV : 23162.711750 ms 35.948413% improvement
TXT with 1/3 escapes: 31786.386000 ms 3.451313% improvement
CSV with 1/3 quotes: 43330.475500 ms 9.029645% improvement
x86 v7-0002
TXT : 22394.812500 ms 23.364552% improvement
CSV : 22374.645750 ms 38.127643% improvement
TXT with 1/3 escapes: 32378.929750 ms 1.651507% improvement
CSV with 1/3 quotes: 47139.171750 ms 1.033461% improvement
arm master
TXT : 9448.900500 ms
CSV : 11135.871500 ms
TXT with 1/3 escapes: 10786.418750 ms
CSV with 1/3 quotes: 14115.335500 ms
arm v7-0001
TXT : 7271.170500 ms 23.047443% improvement
CSV : 7259.866750 ms 34.806479% improvement
TXT with 1/3 escapes: 10894.445500 ms -1.001507% regression
CSV with 1/3 quotes: 13398.444000 ms 5.078813% improvement
arm v7-0002
TXT : 7165.707250 ms 24.163587% improvement
CSV : 7140.497250 ms 35.878416% improvement
TXT with 1/3 escapes: 10308.782250 ms 4.428129% improvement
CSV with 1/3 quotes: 12576.179500 ms 10.904140% improvement
v7-0001 + v7-0002 applied to master certainly seems promising: nice to see
speed improvements across the board on both x86 and arm!
On Fri, Feb 13, 2026 at 5:09 PM Nathan Bossart <nathandbossart@gmail.com>
wrote:
> On Fri, Feb 13, 2026 at 02:45:30PM +0300, Nazir Bilal Yavuz wrote:
> > Also, if I change this code to:
> >
> > if (cstate->simd_enabled)
> > {
> > if (is_csv)
> > result = CopyReadLineText(cstate, true, true);
> > else
> > result = CopyReadLineText(cstate, false, true);
> > }
> > else
> > {
> > if (is_csv)
> > result = CopyReadLineText(cstate, true, false);
> > else
> > result = CopyReadLineText(cstate, false, false);
> > }
> >
> > then I see ~%5 performance improvement in scalar path compared to master.
>
> Hm. What difference do you see if you just do
>
> if (is_csv)
> result = CopyReadLineText(cstate, true);
> else
> result = CopyReadLineText(cstate, false);
>
> both with and without the SIMD stuff? IIUC this is allowing the compiler
> to remove several branches in CopyReadLineText(), which might be a nice
> improvement on its own. That being said, I'm less convinced that adding a
> simd_enabled parameter to CopyReadLineText() helps, because 1) it's
> involved in fewer branches and 2) we change it within the function, so the
> compiler can't remove the branches, anyway. But perhaps I'm missing
> something.
>
> Some other random thoughts:
>
> + match = vector8_or(vector8_eq(chunk, nl),
> vector8_eq(chunk, cr));
>
> + match = vector8_or(vector8_eq(chunk, nl),
> vector8_eq(chunk, cr));
>
> Since \n and \r are well below "normal" ASCII values, I wonder if we could
> simplify these to something like
>
> match = vector8_gt(... vector with all lanes set to \r + 1 ...,
> chunk);
>
> + /* Check if we found any special characters */
> + mask = vector8_highbit_mask(match);
> + if (mask != 0)
>
> vector8_highbit_mask() is somewhat expensive on AArch64, so I wonder if
> waiting until we enter the "if" block to calculate it has any benefit.
>
> + simd_hit_eol = (c1 == '\r' || c1 == '\n') && (!is_csv ||
> !in_quote);
>
> If (is_csv && in_quote), we shouldn't have picked up \r or \n in the first
> place, right?
>
> + simd_hit_eof = c1 == '\\' && c2 == '.' && !is_csv;
> +
> + /*
> + * Do not disable SIMD when we hit EOL or EOF characters.
> In
> + * practice, it does not matter for EOF because parsing
> ends
> + * there, but we keep the behavior consistent.
> + */
> + if (!(simd_hit_eof || simd_hit_eol))
>
> I'd think that doing less unnecessary work would outweigh the benefits of
> consistency for the EOF case.
>
> --
> nathan
>
--
-- Manni Wood EDB: https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 22:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-10 11:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-21 20:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Neil Conway <neil.conway@gmail.com>
2026-01-31 16:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 13:51 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 18:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 21:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-06 22:19 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 22:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-11 13:27 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-11 22:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-13 11:45 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-13 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-14 03:34 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
@ 2026-02-16 17:04 ` Nathan Bossart <nathandbossart@gmail.com>
2026-02-16 18:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
0 siblings, 1 reply; 178+ messages in thread
From: Nathan Bossart @ 2026-02-16 17:04 UTC (permalink / raw)
To: Manni Wood <manni.wood@enterprisedb.com>; +Cc: Nazir Bilal Yavuz <byavuz81@gmail.com>; KAZAR Ayoub <ma_kazar@esi.dz>; Neil Conway <neil.conway@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
On Fri, Feb 13, 2026 at 09:34:13PM -0600, Manni Wood wrote:
> v7-0001 + v7-0002 applied to master certainly seems promising: nice to see
> speed improvements across the board on both x86 and arm!
Thanks for testing. Based on these results, I think we can abandon 0002,
at least for now.
--
nathan
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 22:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-10 11:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-21 20:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Neil Conway <neil.conway@gmail.com>
2026-01-31 16:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 13:51 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 18:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 21:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-06 22:19 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 22:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-11 13:27 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-11 22:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-13 11:45 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-13 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-14 03:34 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-16 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
@ 2026-02-16 18:15 ` Nathan Bossart <nathandbossart@gmail.com>
2026-02-17 05:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
0 siblings, 1 reply; 178+ messages in thread
From: Nathan Bossart @ 2026-02-16 18:15 UTC (permalink / raw)
To: Manni Wood <manni.wood@enterprisedb.com>; +Cc: Nazir Bilal Yavuz <byavuz81@gmail.com>; KAZAR Ayoub <ma_kazar@esi.dz>; Neil Conway <neil.conway@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
On Mon, Feb 16, 2026 at 11:04:58AM -0600, Nathan Bossart wrote:
> On Fri, Feb 13, 2026 at 09:34:13PM -0600, Manni Wood wrote:
>> v7-0001 + v7-0002 applied to master certainly seems promising: nice to see
>> speed improvements across the board on both x86 and arm!
>
> Thanks for testing. Based on these results, I think we can abandon 0002,
> at least for now.
Have you tested small rows, i.e., less than 16 bytes per row? I'm
wondering if that regresses at all.
--
nathan
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 22:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-10 11:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-21 20:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Neil Conway <neil.conway@gmail.com>
2026-01-31 16:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 13:51 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 18:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 21:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-06 22:19 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 22:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-11 13:27 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-11 22:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-13 11:45 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-13 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-14 03:34 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-16 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-16 18:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
@ 2026-02-17 05:01 ` Manni Wood <manni.wood@enterprisedb.com>
2026-02-17 22:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
0 siblings, 1 reply; 178+ messages in thread
From: Manni Wood @ 2026-02-17 05:01 UTC (permalink / raw)
To: Nathan Bossart <nathandbossart@gmail.com>; +Cc: Nazir Bilal Yavuz <byavuz81@gmail.com>; KAZAR Ayoub <ma_kazar@esi.dz>; Neil Conway <neil.conway@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
On Mon, Feb 16, 2026 at 12:15 PM Nathan Bossart <nathandbossart@gmail.com>
wrote:
> On Mon, Feb 16, 2026 at 11:04:58AM -0600, Nathan Bossart wrote:
> > On Fri, Feb 13, 2026 at 09:34:13PM -0600, Manni Wood wrote:
> >> v7-0001 + v7-0002 applied to master certainly seems promising: nice to
> see
> >> speed improvements across the board on both x86 and arm!
> >
> > Thanks for testing. Based on these results, I think we can abandon 0002,
> > at least for now.
>
> Have you tested small rows, i.e., less than 16 bytes per row? I'm
> wondering if that regresses at all.
>
> --
> nathan
>
I ran some tests using narrow rows that look like this:
$ head t_none.txt
BB AA
BB AA
BB AA
$ head t_none.csv
BB,AA
BB,AA
BB,AA
$ head t_escape.txt
B\\B A\\A
B\\B A\\A
B\\B A\\A
$ head t_quote.csv
"B""B","A""A"
"B""B","A""A"
"B""B","A""A"
Here are the results on my x86 tower and my arm raspberry pi 5:
x86 NARROW master copy from
TXT : 2477.022500 ms
CSV : 2825.095500 ms
TXT with 1/3 escapes: 2620.575000 ms
CSV with 1/3 quotes: 3249.058750 ms
x86 NARROW v70001 copy from
TXT : 2475.659000 ms 0.055046% improvement
CSV : 2421.976750 ms 14.269208% improvement
TXT with 1/3 escapes: 2660.953750 ms -1.540836% regression
CSV with 1/3 quotes: 3255.546750 ms -0.199689% regression
x86 NARROW v70002 copy from
TXT : 2481.372250 ms -0.175604% regression
CSV : 2437.541250 ms 13.718271% improvement
TXT with 1/3 escapes: 2646.300000 ms -0.981655% regression
CSV with 1/3 quotes: 3202.014500 ms 1.447935% improvement
arm NARROW master copy from
TXT : 2294.270500 ms
CSV : 2085.839000 ms
TXT with 1/3 escapes: 2467.966000 ms
CSV with 1/3 quotes: 2485.533000 ms
arm NARROW v70001 copy from
TXT : 1982.497500 ms 13.589200% improvement
CSV : 2005.829500 ms 3.835843% improvement
TXT with 1/3 escapes: 2111.778250 ms 14.432442% improvement
CSV with 1/3 quotes: 2441.370000 ms 1.776802% improvement
arm NARROW v70002 copy from
TXT : 1975.982250 ms 13.873179% improvement
CSV : 2022.744000 ms 3.024922% improvement
TXT with 1/3 escapes: 2080.273000 ms 15.709009% improvement
CSV with 1/3 quotes: 2476.819000 ms 0.350589% improvement
Hope this helps!
--
-- Manni Wood EDB: https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 22:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-10 11:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-21 20:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Neil Conway <neil.conway@gmail.com>
2026-01-31 16:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 13:51 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 18:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 21:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-06 22:19 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 22:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-11 13:27 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-11 22:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-13 11:45 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-13 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-14 03:34 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-16 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-16 18:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-17 05:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
@ 2026-02-17 22:29 ` Nathan Bossart <nathandbossart@gmail.com>
0 siblings, 0 replies; 178+ messages in thread
From: Nathan Bossart @ 2026-02-17 22:29 UTC (permalink / raw)
To: Manni Wood <manni.wood@enterprisedb.com>; +Cc: Nazir Bilal Yavuz <byavuz81@gmail.com>; KAZAR Ayoub <ma_kazar@esi.dz>; Neil Conway <neil.conway@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
On Mon, Feb 16, 2026 at 11:01:21PM -0600, Manni Wood wrote:
> Here are the results on my x86 tower and my arm raspberry pi 5:
>
> x86 NARROW master copy from
> TXT : 2477.022500 ms
> CSV : 2825.095500 ms
> TXT with 1/3 escapes: 2620.575000 ms
> CSV with 1/3 quotes: 3249.058750 ms
>
> x86 NARROW v70001 copy from
> TXT : 2475.659000 ms 0.055046% improvement
> CSV : 2421.976750 ms 14.269208% improvement
> TXT with 1/3 escapes: 2660.953750 ms -1.540836% regression
> CSV with 1/3 quotes: 3255.546750 ms -0.199689% regression
>
> x86 NARROW v70002 copy from
> TXT : 2481.372250 ms -0.175604% regression
> CSV : 2437.541250 ms 13.718271% improvement
> TXT with 1/3 escapes: 2646.300000 ms -0.981655% regression
> CSV with 1/3 quotes: 3202.014500 ms 1.447935% improvement
>
>
> arm NARROW master copy from
> TXT : 2294.270500 ms
> CSV : 2085.839000 ms
> TXT with 1/3 escapes: 2467.966000 ms
> CSV with 1/3 quotes: 2485.533000 ms
>
> arm NARROW v70001 copy from
> TXT : 1982.497500 ms 13.589200% improvement
> CSV : 2005.829500 ms 3.835843% improvement
> TXT with 1/3 escapes: 2111.778250 ms 14.432442% improvement
> CSV with 1/3 quotes: 2441.370000 ms 1.776802% improvement
>
> arm NARROW v70002 copy from
> TXT : 1975.982250 ms 13.873179% improvement
> CSV : 2022.744000 ms 3.024922% improvement
> TXT with 1/3 escapes: 2080.273000 ms 15.709009% improvement
> CSV with 1/3 quotes: 2476.819000 ms 0.350589% improvement
Thanks. I don't see anything too terribly concerning here, but we should
still plan to redo these tests once the patch is ready for commit.
--
nathan
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 22:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-10 11:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-21 20:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Neil Conway <neil.conway@gmail.com>
2026-01-31 16:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 13:51 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 18:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 21:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-06 22:19 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 22:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-11 13:27 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-11 22:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-13 11:45 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-13 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
@ 2026-02-18 13:38 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-18 21:26 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2 siblings, 1 reply; 178+ messages in thread
From: Nazir Bilal Yavuz @ 2026-02-18 13:38 UTC (permalink / raw)
To: Nathan Bossart <nathandbossart@gmail.com>; +Cc: KAZAR Ayoub <ma_kazar@esi.dz>; Neil Conway <neil.conway@gmail.com>; Manni Wood <manni.wood@enterprisedb.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
Hi,
On Sat, 14 Feb 2026 at 02:09, Nathan Bossart <nathandbossart@gmail.com> wrote:
>
> On Fri, Feb 13, 2026 at 02:45:30PM +0300, Nazir Bilal Yavuz wrote:
> > Also, if I change this code to:
> >
> > if (cstate->simd_enabled)
> > {
> > if (is_csv)
> > result = CopyReadLineText(cstate, true, true);
> > else
> > result = CopyReadLineText(cstate, false, true);
> > }
> > else
> > {
> > if (is_csv)
> > result = CopyReadLineText(cstate, true, false);
> > else
> > result = CopyReadLineText(cstate, false, false);
> > }
> >
> > then I see ~%5 performance improvement in scalar path compared to master.
>
> Hm. What difference do you see if you just do
>
> if (is_csv)
> result = CopyReadLineText(cstate, true);
> else
> result = CopyReadLineText(cstate, false);
>
> both with and without the SIMD stuff? IIUC this is allowing the compiler
> to remove several branches in CopyReadLineText(), which might be a nice
> improvement on its own. That being said, I'm less convinced that adding a
> simd_enabled parameter to CopyReadLineText() helps, because 1) it's
> involved in fewer branches and 2) we change it within the function, so the
> compiler can't remove the branches, anyway. But perhaps I'm missing
> something.
I did couple of benchmarks, some info about them:
- Benchmarks show percentage comparisons of timings against the master
branch. Positive values mean a regression, while negative values mean
an improvement.
- There are a total 200000 lines in each input and each line is 8192 bytes.
- For the columns, none means there is no special character. The other
numbers represent the ratio of normal characters to special
characters. For example, 0 means all the data is special characters, 4
means %25 of the data is special characters, 16 means 1/16 of the data
is special characters and such.
--------------------
This is the benchmark without the SIMD stuff:
- only_inline: Only change is CopyReadLineText() being inlined.
- is_csv_verbose-wo-inline: is_csv is sent as a constant boolean like
you suggested but CopyReadLineText() isn't inlined.
- is_csv_verbose-w-inline: is_csv is sent as a constant boolean and
CopyReadLineText() is inlined.
+-------------------------------+-------+------+------+-------+-------+------+
| TEXT | none | 0 | 4 | 8 | 16 | 32 |
+-------------------------------+-------+------+------+-------+-------+------+
| only_inline | 0 | +1.6 | 0 | 0 | -1 | 0 |
+-------------------------------+-------+------+------+-------+-------+------+
| is_csv_verbose-wo-inline | 0 | 0 | 0 | 0 | 0 | 0 |
+-------------------------------+-------+------+------+-------+-------+------+
| is_csv_verbose-w-inline | -3.5 | 0 | -7.7 | -2.3 | -4.1 | -3.4 |
+-------------------------------+-------+------+------+-------+-------+------+
| | | | | | | |
+-------------------------------+-------+------+------+-------+-------+------+
| | | | | | | |
+-------------------------------+-------+------+------+-------+-------+------+
| CSV | none | 0 | 4 | 8 | 16 | 32 |
+-------------------------------+-------+------+------+-------+-------+------+
| only_inline | -1.1 | 0 | 0 | 0 | 0 | -1 |
+-------------------------------+-------+------+------+-------+-------+------+
| is_csv_verbose-wo-inline | 0 | 0 | 0 | 0 | -0.3 | 0 |
+-------------------------------+-------+------+------+-------+-------+------+
| is_csv_verbose-w-inline | -4 | -2.3 | -1.2 | -1.8 | -2.8 | -2.7 |
+-------------------------------+-------+------+------+-------+-------+------+
By looking the benchmark results,
if (is_csv)
result = CopyReadLineText(cstate, true);
else
result = CopyReadLineText(cstate, false);
with inline CopyReadLineText() function helps the performance even without SIMD.
----------------------------------------
This is the same benchmark with SIMD stuff:
only_inline: Only change is CopyReadLineText() being inlined, there is
no simd_enabled argument in the CopyReadLineText().
is_csv_verbose-w-inline: is_csv is sent as a constant boolean and
CopyReadLineText() is inlined. There is no simd_enabled argument in
the CopyReadLineText().
simd_enabled_verbose-w-inline: simd_enabled is sent as a constant
boolean and CopyReadLineText() is inlined.
both_verbose_w_inline: both is_csv and simd_enabled are sent as a
constant boolean and CopyReadLineText() is inlined.
+-------------------------------+-------+------+------+-------+-------+------+
| TEXT | none | 0 | 4 | 8 | 16 | 32 |
+-------------------------------+-------+------+------+-------+-------+------+
| only_inline | -11 | +9.7 | +9.1 | +11.4 | +14.8 | +8 |
+-------------------------------+-------+------+------+-------+-------+------+
| is_csv_verbose-wo-inline | -11.9 | +4.5 | +2.4 | +3.5 | +2.2 | +1.8 |
+-------------------------------+-------+------+------+-------+-------+------+
| is_csv_verbose-w-inline | -12.6 | 0 | -2.4 | +2.8 | +1.6 | 0 |
+-------------------------------+-------+------+------+-------+-------+------+
| both_verbose_w_inline | -12.1 | 0 | -5 | 0 | +2.5 | -1.8 |
+-------------------------------+-------+------+------+-------+-------+------+
| | | | | | | |
+-------------------------------+-------+------+------+-------+-------+------+
| CSV | none | 0 | 4 | 8 | 16 | 32 |
+-------------------------------+-------+------+------+-------+-------+------+
| only_inline | -22.6 | +4.2 | +2.1 | +2.6 | 0 | +2.2 |
+-------------------------------+-------+------+------+-------+-------+------+
| is_csv_verbose-w-inline | -22.5 | -2.1 | -3.4 | -3.9 | -6.4 | -3.4 |
+-------------------------------+-------+------+------+-------+-------+------+
| simd_enabled_verbose-w-inline | -23 | 0 | -1.9 | -2.2 | -4.8 | -1.6 |
+-------------------------------+-------+------+------+-------+-------+------+
| both_verbose_w_inline | -23.3 | -2.9 | -5 | -4.5 | -7.1 | -4.3 |
+-------------------------------+-------+------+------+-------+-------+------+
By looking at these results having both is_csv and simd_enabled as an
argument and sending them as constant boolean arguments help most.
>
> Some other random thoughts:
I am sending the benchmark results for now. I haven't tested other
suggestions yet. I will follow up with another email once I have
tested them.
--
Regards,
Nazir Bilal Yavuz
Microsoft
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 22:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-10 11:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-21 20:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Neil Conway <neil.conway@gmail.com>
2026-01-31 16:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 13:51 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 18:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 21:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-06 22:19 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 22:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-11 13:27 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-11 22:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-13 11:45 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-13 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 13:38 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
@ 2026-02-18 21:26 ` Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 21:54 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
0 siblings, 1 reply; 178+ messages in thread
From: Nathan Bossart @ 2026-02-18 21:26 UTC (permalink / raw)
To: Nazir Bilal Yavuz <byavuz81@gmail.com>; +Cc: KAZAR Ayoub <ma_kazar@esi.dz>; Neil Conway <neil.conway@gmail.com>; Manni Wood <manni.wood@enterprisedb.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
On Wed, Feb 18, 2026 at 04:38:07PM +0300, Nazir Bilal Yavuz wrote:
> By looking at these results having both is_csv and simd_enabled as an
> argument and sending them as constant boolean arguments help most.
Thanks for doing these tests. ISTM we might as well get this initial
inlining stuff committed. Thoughts?
--
nathan
From 516176894a682e8bd4cb32202e3726f6621ba44d Mon Sep 17 00:00:00 2001
From: Nathan Bossart <nathan@postgresql.org>
Date: Wed, 18 Feb 2026 14:55:39 -0600
Subject: [PATCH v8 1/1] Speedup COPY FROM with additional function inlining.
Following the example set by commit 58a359e585, we can squeeze out
a little more performance from COPY FROM (FORMAT {text,csv}) by
forcing CopyReadLineText() to be inlined and by passing the is_csv
parameter as a constant. This allows the compiler to emit
specialized code with the known-const false comparisons and
subsequent branches removed.
Author: Nazir Bilal Yavuz <byavuz81@gmail.com>
Reviewed-by: Ayoub Kazar <ma_kazar@esi.dz>
Discussion: https://postgr.es/m/CAOzEurSW8cNr6TPKsjrstnPfhf4QyQqB4tnPXGGe8N4e_v7Jig%40mail.gmail.com
---
src/backend/commands/copyfromparse.c | 19 +++++++++++++++----
1 file changed, 15 insertions(+), 4 deletions(-)
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index 94d6f415a06..0aa549837b5 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -141,7 +141,8 @@ static const char BinarySignature[11] = "PGCOPY\n\377\r\n\0";
/* non-export function prototypes */
static bool CopyReadLine(CopyFromState cstate, bool is_csv);
-static bool CopyReadLineText(CopyFromState cstate, bool is_csv);
+static pg_attribute_always_inline bool CopyReadLineText(CopyFromState cstate,
+ bool is_csv);
static int CopyReadAttributesText(CopyFromState cstate);
static int CopyReadAttributesCSV(CopyFromState cstate);
static Datum CopyReadBinaryAttribute(CopyFromState cstate, FmgrInfo *flinfo,
@@ -1173,8 +1174,18 @@ CopyReadLine(CopyFromState cstate, bool is_csv)
resetStringInfo(&cstate->line_buf);
cstate->line_buf_valid = false;
- /* Parse data and transfer into line_buf */
- result = CopyReadLineText(cstate, is_csv);
+ /*
+ * Parse data and transfer into line_buf.
+ *
+ * Because this is performance critical, we inline CopyReadLineText() and
+ * pass the boolean parameters as constants to allow the compiler to emit
+ * specialized code with the known-const false comparisons and subsequent
+ * branches removed.
+ */
+ if (is_csv)
+ result = CopyReadLineText(cstate, true);
+ else
+ result = CopyReadLineText(cstate, false);
if (result)
{
@@ -1241,7 +1252,7 @@ CopyReadLine(CopyFromState cstate, bool is_csv)
/*
* CopyReadLineText - inner loop of CopyReadLine for text mode
*/
-static bool
+static pg_attribute_always_inline bool
CopyReadLineText(CopyFromState cstate, bool is_csv)
{
char *copy_input_buf;
--
2.50.1 (Apple Git-155)
Attachments:
[text/plain] v8-0001-Speedup-COPY-FROM-with-additional-function-inlini.patch (2.6K, ../../aZYudtuBLVb36pZE@nathan/2-v8-0001-Speedup-COPY-FROM-with-additional-function-inlini.patch)
download | inline diff:
From 516176894a682e8bd4cb32202e3726f6621ba44d Mon Sep 17 00:00:00 2001
From: Nathan Bossart <nathan@postgresql.org>
Date: Wed, 18 Feb 2026 14:55:39 -0600
Subject: [PATCH v8 1/1] Speedup COPY FROM with additional function inlining.
Following the example set by commit 58a359e585, we can squeeze out
a little more performance from COPY FROM (FORMAT {text,csv}) by
forcing CopyReadLineText() to be inlined and by passing the is_csv
parameter as a constant. This allows the compiler to emit
specialized code with the known-const false comparisons and
subsequent branches removed.
Author: Nazir Bilal Yavuz <byavuz81@gmail.com>
Reviewed-by: Ayoub Kazar <ma_kazar@esi.dz>
Discussion: https://postgr.es/m/CAOzEurSW8cNr6TPKsjrstnPfhf4QyQqB4tnPXGGe8N4e_v7Jig%40mail.gmail.com
---
src/backend/commands/copyfromparse.c | 19 +++++++++++++++----
1 file changed, 15 insertions(+), 4 deletions(-)
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index 94d6f415a06..0aa549837b5 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -141,7 +141,8 @@ static const char BinarySignature[11] = "PGCOPY\n\377\r\n\0";
/* non-export function prototypes */
static bool CopyReadLine(CopyFromState cstate, bool is_csv);
-static bool CopyReadLineText(CopyFromState cstate, bool is_csv);
+static pg_attribute_always_inline bool CopyReadLineText(CopyFromState cstate,
+ bool is_csv);
static int CopyReadAttributesText(CopyFromState cstate);
static int CopyReadAttributesCSV(CopyFromState cstate);
static Datum CopyReadBinaryAttribute(CopyFromState cstate, FmgrInfo *flinfo,
@@ -1173,8 +1174,18 @@ CopyReadLine(CopyFromState cstate, bool is_csv)
resetStringInfo(&cstate->line_buf);
cstate->line_buf_valid = false;
- /* Parse data and transfer into line_buf */
- result = CopyReadLineText(cstate, is_csv);
+ /*
+ * Parse data and transfer into line_buf.
+ *
+ * Because this is performance critical, we inline CopyReadLineText() and
+ * pass the boolean parameters as constants to allow the compiler to emit
+ * specialized code with the known-const false comparisons and subsequent
+ * branches removed.
+ */
+ if (is_csv)
+ result = CopyReadLineText(cstate, true);
+ else
+ result = CopyReadLineText(cstate, false);
if (result)
{
@@ -1241,7 +1252,7 @@ CopyReadLine(CopyFromState cstate, bool is_csv)
/*
* CopyReadLineText - inner loop of CopyReadLine for text mode
*/
-static bool
+static pg_attribute_always_inline bool
CopyReadLineText(CopyFromState cstate, bool is_csv)
{
char *copy_input_buf;
--
2.50.1 (Apple Git-155)
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 22:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-10 11:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-21 20:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Neil Conway <neil.conway@gmail.com>
2026-01-31 16:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 13:51 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 18:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 21:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-06 22:19 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 22:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-11 13:27 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-11 22:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-13 11:45 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-13 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 13:38 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-18 21:26 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
@ 2026-02-18 21:54 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 04:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
0 siblings, 1 reply; 178+ messages in thread
From: Nazir Bilal Yavuz @ 2026-02-18 21:54 UTC (permalink / raw)
To: Nathan Bossart <nathandbossart@gmail.com>; +Cc: KAZAR Ayoub <ma_kazar@esi.dz>; Neil Conway <neil.conway@gmail.com>; Manni Wood <manni.wood@enterprisedb.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
Hi,
On Thu, 19 Feb 2026 at 00:26, Nathan Bossart <nathandbossart@gmail.com> wrote:
>
> On Wed, Feb 18, 2026 at 04:38:07PM +0300, Nazir Bilal Yavuz wrote:
> > By looking at these results having both is_csv and simd_enabled as an
> > argument and sending them as constant boolean arguments help most.
>
> Thanks for doing these tests. ISTM we might as well get this initial
> inlining stuff committed. Thoughts?
nitpick:
-static bool CopyReadLineText(CopyFromState cstate, bool is_csv);
+static pg_attribute_always_inline bool CopyReadLineText(CopyFromState cstate,
+ bool is_csv);
Do we want to move the new CopyReadLineText() declaration below to
group it with the other functions marked pg_attribute_always_inline?
Other than that, LGTM. I think it makes sense to separately commit this.
--
Regards,
Nazir Bilal Yavuz
Microsoft
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 22:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-10 11:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-21 20:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Neil Conway <neil.conway@gmail.com>
2026-01-31 16:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 13:51 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 18:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 21:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-06 22:19 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 22:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-11 13:27 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-11 22:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-13 11:45 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-13 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 13:38 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-18 21:26 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 21:54 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
@ 2026-02-19 04:02 ` Manni Wood <manni.wood@enterprisedb.com>
2026-02-19 09:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
0 siblings, 1 reply; 178+ messages in thread
From: Manni Wood @ 2026-02-19 04:02 UTC (permalink / raw)
To: Nazir Bilal Yavuz <byavuz81@gmail.com>; +Cc: Nathan Bossart <nathandbossart@gmail.com>; KAZAR Ayoub <ma_kazar@esi.dz>; Neil Conway <neil.conway@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
On Wed, Feb 18, 2026 at 3:54 PM Nazir Bilal Yavuz <byavuz81@gmail.com>
wrote:
> Hi,
>
> On Thu, 19 Feb 2026 at 00:26, Nathan Bossart <nathandbossart@gmail.com>
> wrote:
> >
> > On Wed, Feb 18, 2026 at 04:38:07PM +0300, Nazir Bilal Yavuz wrote:
> > > By looking at these results having both is_csv and simd_enabled as an
> > > argument and sending them as constant boolean arguments help most.
> >
> > Thanks for doing these tests. ISTM we might as well get this initial
> > inlining stuff committed. Thoughts?
>
> nitpick:
>
> -static bool CopyReadLineText(CopyFromState cstate, bool is_csv);
> +static pg_attribute_always_inline bool CopyReadLineText(CopyFromState
> cstate,
> + bool is_csv);
>
> Do we want to move the new CopyReadLineText() declaration below to
> group it with the other functions marked pg_attribute_always_inline?
>
> Other than that, LGTM. I think it makes sense to separately commit this.
>
> --
> Regards,
> Nazir Bilal Yavuz
> Microsoft
>
Hello.
I took some time tonight to apply v8 to the latest master (759b03b2) on my
x86 tower and arm raspberry pi 5.
Here are the results, using both narrow columns and the wider columns we've
been using througout:
x86 master NARROW
TXT : 2587.642000 ms
CSV : 2621.759000 ms
TXT with 1/3 escapes: 2707.933500 ms
CSV with 1/3 quotes: 3254.896500 ms
x86 v8 NARROW
TXT : 2488.655250 ms 3.825365% improvement
CSV : 2628.818000 ms -0.269247% regression
TXT with 1/3 escapes: 2615.522000 ms 3.412621% improvement
CSV with 1/3 quotes: 3446.368000 ms -5.882568% regression
x86 master WIDE
TXT : 30583.229500 ms
CSV : 35054.533500 ms
TXT with 1/3 escapes: 32767.421500 ms
CSV with 1/3 quotes: 44214.163500 ms
x86 v8 WIDE
TXT : 26527.494250 ms 13.261305% improvement
CSV : 33364.443750 ms 4.821316% improvement
TXT with 1/3 escapes: 29320.648000 ms 10.518904% improvement
CSV with 1/3 quotes: 42334.074750 ms 4.252232% improvement
arm master NARROW
TXT : 1999.401000 ms
CSV : 2081.610750 ms
TXT with 1/3 escapes: 2053.230250 ms
CSV with 1/3 quotes: 2431.608750 ms
arm v8 NARROW
TXT : 1981.663750 ms 0.887128% improvement
CSV : 2023.892500 ms 2.772769% improvement
TXT with 1/3 escapes: 2004.215250 ms 2.387214% improvement
CSV with 1/3 quotes: 2616.872750 ms -7.618989% regression
arm master WIDE
TXT : 9120.731750 ms
CSV : 11114.478250 ms
TXT with 1/3 escapes: 10338.124500 ms
CSV with 1/3 quotes: 13404.430250 ms
arm v8 WIDE
TXT : 8430.090750 ms 7.572210% improvement
CSV : 10115.135500 ms 8.991360% improvement
TXT with 1/3 escapes: 9624.383500 ms 6.903970% improvement
CSV with 1/3 quotes: 12331.714000 ms 8.002699% improvement
--
-- Manni Wood EDB: https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 22:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-10 11:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-21 20:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Neil Conway <neil.conway@gmail.com>
2026-01-31 16:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 13:51 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 18:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 21:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-06 22:19 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 22:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-11 13:27 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-11 22:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-13 11:45 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-13 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 13:38 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-18 21:26 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 21:54 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 04:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
@ 2026-02-19 09:01 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 22:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
0 siblings, 1 reply; 178+ messages in thread
From: Nazir Bilal Yavuz @ 2026-02-19 09:01 UTC (permalink / raw)
To: Manni Wood <manni.wood@enterprisedb.com>; +Cc: Nathan Bossart <nathandbossart@gmail.com>; KAZAR Ayoub <ma_kazar@esi.dz>; Neil Conway <neil.conway@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
Hi,
On Thu, 19 Feb 2026 at 07:02, Manni Wood <manni.wood@enterprisedb.com> wrote:
>
> I took some time tonight to apply v8 to the latest master (759b03b2) on my x86 tower and arm raspberry pi 5.
>
> Here are the results, using both narrow columns and the wider columns we've been using througout:
>
> x86 master NARROW
> TXT : 2587.642000 ms
> CSV : 2621.759000 ms
> TXT with 1/3 escapes: 2707.933500 ms
> CSV with 1/3 quotes: 3254.896500 ms
>
> x86 v8 NARROW
> TXT : 2488.655250 ms 3.825365% improvement
> CSV : 2628.818000 ms -0.269247% regression
> TXT with 1/3 escapes: 2615.522000 ms 3.412621% improvement
> CSV with 1/3 quotes: 3446.368000 ms -5.882568% regression
>
> x86 master WIDE
> TXT : 30583.229500 ms
> CSV : 35054.533500 ms
> TXT with 1/3 escapes: 32767.421500 ms
> CSV with 1/3 quotes: 44214.163500 ms
>
> x86 v8 WIDE
> TXT : 26527.494250 ms 13.261305% improvement
> CSV : 33364.443750 ms 4.821316% improvement
> TXT with 1/3 escapes: 29320.648000 ms 10.518904% improvement
> CSV with 1/3 quotes: 42334.074750 ms 4.252232% improvement
>
>
>
> arm master NARROW
> TXT : 1999.401000 ms
> CSV : 2081.610750 ms
> TXT with 1/3 escapes: 2053.230250 ms
> CSV with 1/3 quotes: 2431.608750 ms
>
> arm v8 NARROW
> TXT : 1981.663750 ms 0.887128% improvement
> CSV : 2023.892500 ms 2.772769% improvement
> TXT with 1/3 escapes: 2004.215250 ms 2.387214% improvement
> CSV with 1/3 quotes: 2616.872750 ms -7.618989% regression
>
> arm master WIDE
> TXT : 9120.731750 ms
> CSV : 11114.478250 ms
> TXT with 1/3 escapes: 10338.124500 ms
> CSV with 1/3 quotes: 13404.430250 ms
>
> arm v8 WIDE
> TXT : 8430.090750 ms 7.572210% improvement
> CSV : 10115.135500 ms 8.991360% improvement
> TXT with 1/3 escapes: 9624.383500 ms 6.903970% improvement
> CSV with 1/3 quotes: 12331.714000 ms 8.002699% improvement
Thank you for the results, they are interesting. I didn't expect to
see any regression for this benchmark. Also, I would expect the
non-special character cases and the 1/3 special character cases to
perform similarly, since we are not using SIMD for this benchmark.
I noticed that the timings in your narrow benchmark (both x86 and ARM)
are quite short. Would it be possible to extend the test so that the
total runtime is closer to ~10,000 ms? That might give us more stable
results.
Here is my benchmark with using your script:
WIDE: Total 500000 lines and each line is 4096 bytes.
NARROW: Total 1500000 lines and each line is 2-4 bytes (`"A""A"` and `A\\A`).
+---------+---------------+---------------+---------------+----------------+
| WIDE | TXT None | TXT 1/3 | CSV None | CSV 1/3 |
+---------+---------------+---------------+---------------+----------------+
| master | 10512 | 11133 | 12241 | 14321 |
+---------+---------------+---------------+---------------+----------------+
| patched | 10000 (-%4.8) | 10804 (-%2.9) | 11571 (-%5.4) | 14008 (-%2.18) |
+---------+---------------+---------------+---------------+----------------+
| | | | | |
+---------+---------------+---------------+---------------+----------------+
| NARROW | | | | |
+---------+---------------+---------------+---------------+----------------+
| master | 9702 | 9745 | 9784 | 10149 |
+---------+---------------+---------------+---------------+----------------+
| patched | 9344 (-%3.6) | 9477 (-%2.7) | 9439 (-%3.5) | 9751 (-%3.9) |
+---------+---------------+---------------+---------------+----------------+
The results look promising to me.
--
Regards,
Nazir Bilal Yavuz
Microsoft
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 22:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-10 11:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-21 20:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Neil Conway <neil.conway@gmail.com>
2026-01-31 16:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 13:51 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 18:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 21:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-06 22:19 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 22:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-11 13:27 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-11 22:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-13 11:45 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-13 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 13:38 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-18 21:26 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 21:54 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 04:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-19 09:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
@ 2026-02-19 22:37 ` KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-20 00:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
0 siblings, 1 reply; 178+ messages in thread
From: KAZAR Ayoub @ 2026-02-19 22:37 UTC (permalink / raw)
To: Nazir Bilal Yavuz <byavuz81@gmail.com>; +Cc: Manni Wood <manni.wood@enterprisedb.com>; Nathan Bossart <nathandbossart@gmail.com>; Neil Conway <neil.conway@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
Hello,
I ran some long benchmarks on this, and I got stable results across
multiple runs (few milliseconds difference)
This is on an Intel I7-1255U CPU with:
sudo cpupower frequency-set --governor=performance
sudo cpupower idle-set -D 0
echo "1" | sudo tee /sys/devices/system/cpu/intel_pstate/no_turbo
WIDE (500k rows)
TXT | none
Master avg: 22,183 ms
New avg: 20,435 ms
Improvement: -7.88%
CSV | none
Master avg: 26,737 ms
New avg: 24,625 ms
Improvement: -7.90%
TXT | escape
Master avg: 26,720 ms
New avg: 23,658 ms
Improvement: -11.46%
CSV | quote
Master avg: 35,961 ms
New avg: 33,317 ms
Improvement: -7.35%
--------------------------------------
NARROW (1.5M rows)
TXT | none
Master avg: 2,220 ms
New avg: 2,125 ms
Improvement: -4.28%
CSV | none
Master avg: 2,330 ms
New avg: 2,145 ms
Improvement: -7.92%
TXT | escape
Master avg: 2,425 ms
New avg: 2,187 ms
Improvement: -9.79%
CSV | quote
Master avg: 2,272 ms
New avg: 2,253 ms
Improvement: -0.85%
No regressions as expected, overall this looks good.
Regards,
Ayoub
On Thu, Feb 19, 2026 at 10:01 AM Nazir Bilal Yavuz <byavuz81@gmail.com>
wrote:
> Hi,
>
> On Thu, 19 Feb 2026 at 07:02, Manni Wood <manni.wood@enterprisedb.com>
> wrote:
> >
> > I took some time tonight to apply v8 to the latest master (759b03b2) on
> my x86 tower and arm raspberry pi 5.
> >
> > Here are the results, using both narrow columns and the wider columns
> we've been using througout:
> >
> > x86 master NARROW
> > TXT : 2587.642000 ms
> > CSV : 2621.759000 ms
> > TXT with 1/3 escapes: 2707.933500 ms
> > CSV with 1/3 quotes: 3254.896500 ms
> >
> > x86 v8 NARROW
> > TXT : 2488.655250 ms 3.825365% improvement
> > CSV : 2628.818000 ms -0.269247% regression
> > TXT with 1/3 escapes: 2615.522000 ms 3.412621% improvement
> > CSV with 1/3 quotes: 3446.368000 ms -5.882568% regression
> >
> > x86 master WIDE
> > TXT : 30583.229500 ms
> > CSV : 35054.533500 ms
> > TXT with 1/3 escapes: 32767.421500 ms
> > CSV with 1/3 quotes: 44214.163500 ms
> >
> > x86 v8 WIDE
> > TXT : 26527.494250 ms 13.261305% improvement
> > CSV : 33364.443750 ms 4.821316% improvement
> > TXT with 1/3 escapes: 29320.648000 ms 10.518904% improvement
> > CSV with 1/3 quotes: 42334.074750 ms 4.252232% improvement
> >
> >
> >
> > arm master NARROW
> > TXT : 1999.401000 ms
> > CSV : 2081.610750 ms
> > TXT with 1/3 escapes: 2053.230250 ms
> > CSV with 1/3 quotes: 2431.608750 ms
> >
> > arm v8 NARROW
> > TXT : 1981.663750 ms 0.887128% improvement
> > CSV : 2023.892500 ms 2.772769% improvement
> > TXT with 1/3 escapes: 2004.215250 ms 2.387214% improvement
> > CSV with 1/3 quotes: 2616.872750 ms -7.618989% regression
> >
> > arm master WIDE
> > TXT : 9120.731750 ms
> > CSV : 11114.478250 ms
> > TXT with 1/3 escapes: 10338.124500 ms
> > CSV with 1/3 quotes: 13404.430250 ms
> >
> > arm v8 WIDE
> > TXT : 8430.090750 ms 7.572210% improvement
> > CSV : 10115.135500 ms 8.991360% improvement
> > TXT with 1/3 escapes: 9624.383500 ms 6.903970% improvement
> > CSV with 1/3 quotes: 12331.714000 ms 8.002699% improvement
>
> Thank you for the results, they are interesting. I didn't expect to
> see any regression for this benchmark. Also, I would expect the
> non-special character cases and the 1/3 special character cases to
> perform similarly, since we are not using SIMD for this benchmark.
>
> I noticed that the timings in your narrow benchmark (both x86 and ARM)
> are quite short. Would it be possible to extend the test so that the
> total runtime is closer to ~10,000 ms? That might give us more stable
> results.
>
> Here is my benchmark with using your script:
>
> WIDE: Total 500000 lines and each line is 4096 bytes.
> NARROW: Total 1500000 lines and each line is 2-4 bytes (`"A""A"` and
> `A\\A`).
>
>
> +---------+---------------+---------------+---------------+----------------+
> | WIDE | TXT None | TXT 1/3 | CSV None | CSV 1/3
> |
>
> +---------+---------------+---------------+---------------+----------------+
> | master | 10512 | 11133 | 12241 | 14321
> |
>
> +---------+---------------+---------------+---------------+----------------+
> | patched | 10000 (-%4.8) | 10804 (-%2.9) | 11571 (-%5.4) | 14008 (-%2.18)
> |
>
> +---------+---------------+---------------+---------------+----------------+
> | | | | |
> |
>
> +---------+---------------+---------------+---------------+----------------+
> | NARROW | | | |
> |
>
> +---------+---------------+---------------+---------------+----------------+
> | master | 9702 | 9745 | 9784 | 10149
> |
>
> +---------+---------------+---------------+---------------+----------------+
> | patched | 9344 (-%3.6) | 9477 (-%2.7) | 9439 (-%3.5) | 9751 (-%3.9)
> |
>
> +---------+---------------+---------------+---------------+----------------+
>
> The results look promising to me.
>
> --
> Regards,
> Nazir Bilal Yavuz
> Microsoft
>
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 22:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-10 11:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-21 20:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Neil Conway <neil.conway@gmail.com>
2026-01-31 16:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 13:51 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 18:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 21:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-06 22:19 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 22:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-11 13:27 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-11 22:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-13 11:45 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-13 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 13:38 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-18 21:26 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 21:54 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 04:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-19 09:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 22:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
@ 2026-02-20 00:09 ` Manni Wood <manni.wood@enterprisedb.com>
2026-02-20 09:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
0 siblings, 1 reply; 178+ messages in thread
From: Manni Wood @ 2026-02-20 00:09 UTC (permalink / raw)
To: KAZAR Ayoub <ma_kazar@esi.dz>; +Cc: Nazir Bilal Yavuz <byavuz81@gmail.com>; Nathan Bossart <nathandbossart@gmail.com>; Neil Conway <neil.conway@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
On Thu, Feb 19, 2026 at 4:37 PM KAZAR Ayoub <ma_kazar@esi.dz> wrote:
> Hello,
>
> I ran some long benchmarks on this, and I got stable results across
> multiple runs (few milliseconds difference)
>
> This is on an Intel I7-1255U CPU with:
> sudo cpupower frequency-set --governor=performance
> sudo cpupower idle-set -D 0
> echo "1" | sudo tee /sys/devices/system/cpu/intel_pstate/no_turbo
>
> WIDE (500k rows)
>
> TXT | none
> Master avg: 22,183 ms
> New avg: 20,435 ms
> Improvement: -7.88%
>
> CSV | none
> Master avg: 26,737 ms
> New avg: 24,625 ms
> Improvement: -7.90%
>
> TXT | escape
> Master avg: 26,720 ms
> New avg: 23,658 ms
> Improvement: -11.46%
>
> CSV | quote
> Master avg: 35,961 ms
> New avg: 33,317 ms
> Improvement: -7.35%
>
> --------------------------------------
>
> NARROW (1.5M rows)
>
> TXT | none
> Master avg: 2,220 ms
> New avg: 2,125 ms
> Improvement: -4.28%
>
> CSV | none
> Master avg: 2,330 ms
> New avg: 2,145 ms
> Improvement: -7.92%
>
> TXT | escape
> Master avg: 2,425 ms
> New avg: 2,187 ms
> Improvement: -9.79%
>
> CSV | quote
> Master avg: 2,272 ms
> New avg: 2,253 ms
> Improvement: -0.85%
>
> No regressions as expected, overall this looks good.
>
> Regards,
>
> Ayoub
>
> On Thu, Feb 19, 2026 at 10:01 AM Nazir Bilal Yavuz <byavuz81@gmail.com>
> wrote:
>
>> Hi,
>>
>> On Thu, 19 Feb 2026 at 07:02, Manni Wood <manni.wood@enterprisedb.com>
>> wrote:
>> >
>> > I took some time tonight to apply v8 to the latest master (759b03b2) on
>> my x86 tower and arm raspberry pi 5.
>> >
>> > Here are the results, using both narrow columns and the wider columns
>> we've been using througout:
>> >
>> > x86 master NARROW
>> > TXT : 2587.642000 ms
>> > CSV : 2621.759000 ms
>> > TXT with 1/3 escapes: 2707.933500 ms
>> > CSV with 1/3 quotes: 3254.896500 ms
>> >
>> > x86 v8 NARROW
>> > TXT : 2488.655250 ms 3.825365% improvement
>> > CSV : 2628.818000 ms -0.269247% regression
>> > TXT with 1/3 escapes: 2615.522000 ms 3.412621% improvement
>> > CSV with 1/3 quotes: 3446.368000 ms -5.882568% regression
>> >
>> > x86 master WIDE
>> > TXT : 30583.229500 ms
>> > CSV : 35054.533500 ms
>> > TXT with 1/3 escapes: 32767.421500 ms
>> > CSV with 1/3 quotes: 44214.163500 ms
>> >
>> > x86 v8 WIDE
>> > TXT : 26527.494250 ms 13.261305% improvement
>> > CSV : 33364.443750 ms 4.821316% improvement
>> > TXT with 1/3 escapes: 29320.648000 ms 10.518904% improvement
>> > CSV with 1/3 quotes: 42334.074750 ms 4.252232% improvement
>> >
>> >
>> >
>> > arm master NARROW
>> > TXT : 1999.401000 ms
>> > CSV : 2081.610750 ms
>> > TXT with 1/3 escapes: 2053.230250 ms
>> > CSV with 1/3 quotes: 2431.608750 ms
>> >
>> > arm v8 NARROW
>> > TXT : 1981.663750 ms 0.887128% improvement
>> > CSV : 2023.892500 ms 2.772769% improvement
>> > TXT with 1/3 escapes: 2004.215250 ms 2.387214% improvement
>> > CSV with 1/3 quotes: 2616.872750 ms -7.618989% regression
>> >
>> > arm master WIDE
>> > TXT : 9120.731750 ms
>> > CSV : 11114.478250 ms
>> > TXT with 1/3 escapes: 10338.124500 ms
>> > CSV with 1/3 quotes: 13404.430250 ms
>> >
>> > arm v8 WIDE
>> > TXT : 8430.090750 ms 7.572210% improvement
>> > CSV : 10115.135500 ms 8.991360% improvement
>> > TXT with 1/3 escapes: 9624.383500 ms 6.903970% improvement
>> > CSV with 1/3 quotes: 12331.714000 ms 8.002699% improvement
>>
>> Thank you for the results, they are interesting. I didn't expect to
>> see any regression for this benchmark. Also, I would expect the
>> non-special character cases and the 1/3 special character cases to
>> perform similarly, since we are not using SIMD for this benchmark.
>>
>> I noticed that the timings in your narrow benchmark (both x86 and ARM)
>> are quite short. Would it be possible to extend the test so that the
>> total runtime is closer to ~10,000 ms? That might give us more stable
>> results.
>>
>> Here is my benchmark with using your script:
>>
>> WIDE: Total 500000 lines and each line is 4096 bytes.
>> NARROW: Total 1500000 lines and each line is 2-4 bytes (`"A""A"` and
>> `A\\A`).
>>
>>
>> +---------+---------------+---------------+---------------+----------------+
>> | WIDE | TXT None | TXT 1/3 | CSV None | CSV 1/3
>> |
>>
>> +---------+---------------+---------------+---------------+----------------+
>> | master | 10512 | 11133 | 12241 | 14321
>> |
>>
>> +---------+---------------+---------------+---------------+----------------+
>> | patched | 10000 (-%4.8) | 10804 (-%2.9) | 11571 (-%5.4) | 14008
>> (-%2.18) |
>>
>> +---------+---------------+---------------+---------------+----------------+
>> | | | | |
>> |
>>
>> +---------+---------------+---------------+---------------+----------------+
>> | NARROW | | | |
>> |
>>
>> +---------+---------------+---------------+---------------+----------------+
>> | master | 9702 | 9745 | 9784 | 10149
>> |
>>
>> +---------+---------------+---------------+---------------+----------------+
>> | patched | 9344 (-%3.6) | 9477 (-%2.7) | 9439 (-%3.5) | 9751 (-%3.9)
>> |
>>
>> +---------+---------------+---------------+---------------+----------------+
>>
>> The results look promising to me.
>>
>> --
>> Regards,
>> Nazir Bilal Yavuz
>> Microsoft
>>
>
Hello!
Thanks for running benchmarks, Ayoub.
Nazir, I ran my benchmarks with more rows this time --- as many rows as
would fit on my test computers without exhausting their RAM disks. That
seems to have brought things more into line with what Ayoub saw. I did get
some small regressions, but I suspect those are not a big deal. (For
instance, on both machines I also noticed the occasional "truncate table"
would take longer than the others, despite my scripts' best efforts to
steady a CPU core and pin postmaster and children to that core.)
x86 WIDE master 500,000 rows
TXT : 30602.244000 ms
CSV : 35062.451250 ms
TXT with 1/3 escapes: 32704.250250 ms
CSV with 1/3 quotes: 44128.072500 ms
x86 WIDE v8 500,000 rows
TXT : 26611.953250 ms 13.039210% improvement
CSV : 33366.184000 ms 4.837846% improvement
TXT with 1/3 escapes: 29251.310000 ms 10.558078% improvement
CSV with 1/3 quotes: 42368.421000 ms 3.987601% improvement
x86 NARROW master 50mil rows
TXT : 25898.004000 ms
CSV : 27212.684500 ms
TXT with 1/3 escapes: 29189.518250 ms
CSV with 1/3 quotes: 33222.510250 ms
x86 NARROW v8 50mil rows
TXT : 26368.765000 ms -1.817750% regression
CSV : 26711.122250 ms 1.843119% improvement
TXT with 1/3 escapes: 28081.150750 ms 3.797142% improvement
CSV with 1/3 quotes: 32851.963500 ms 1.115348% improvement
arm WIDE master 250,000 rows
TXT : 11392.462750 ms
CSV : 13887.576500 ms
TXT with 1/3 escapes: 12908.560750 ms
CSV with 1/3 quotes: 16699.337000 ms
arm WIDE v8 250,000 rows
TXT : 10524.567750 ms 7.618151% improvement
CSV : 12621.211250 ms 9.118691% improvement
TXT with 1/3 escapes: 12017.030250 ms 6.906506% improvement
CSV with 1/3 quotes: 15428.020500 ms 7.612976% improvement
arm NARROW master 25mil rows
TXT : 10030.274000 ms
CSV : 10245.238750 ms
TXT with 1/3 escapes: 10345.224500 ms
CSV with 1/3 quotes: 12186.313250 ms
arm NARROW v8 25mil rows
TXT : 10197.386500 ms -1.666081% regression
CSV : 10257.918750 ms -0.123765% regression
TXT with 1/3 escapes: 10084.978500 ms 2.515615% improvement
CSV with 1/3 quotes: 12064.215000 ms 1.001929% improvement
--
-- Manni Wood EDB: https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 22:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-10 11:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-21 20:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Neil Conway <neil.conway@gmail.com>
2026-01-31 16:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 13:51 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 18:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 21:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-06 22:19 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 22:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-11 13:27 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-11 22:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-13 11:45 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-13 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 13:38 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-18 21:26 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 21:54 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 04:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-19 09:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 22:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-20 00:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
@ 2026-02-20 09:50 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-20 18:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
0 siblings, 1 reply; 178+ messages in thread
From: Nazir Bilal Yavuz @ 2026-02-20 09:50 UTC (permalink / raw)
To: Manni Wood <manni.wood@enterprisedb.com>; +Cc: KAZAR Ayoub <ma_kazar@esi.dz>; Nathan Bossart <nathandbossart@gmail.com>; Neil Conway <neil.conway@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
Hi,
On Fri, 20 Feb 2026 at 03:09, Manni Wood <manni.wood@enterprisedb.com> wrote:
>
> Thanks for running benchmarks, Ayoub.
>
> Nazir, I ran my benchmarks with more rows this time --- as many rows as would fit on my test computers without exhausting their RAM disks. That seems to have brought things more into line with what Ayoub saw. I did get some small regressions, but I suspect those are not a big deal. (For instance, on both machines I also noticed the occasional "truncate table" would take longer than the others, despite my scripts' best efforts to steady a CPU core and pin postmaster and children to that core.)
Thank you both for the benchmarks. Results look good to me!
--
Regards,
Nazir Bilal Yavuz
Microsoft
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 22:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-10 11:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-21 20:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Neil Conway <neil.conway@gmail.com>
2026-01-31 16:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 13:51 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 18:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 21:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-06 22:19 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 22:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-11 13:27 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-11 22:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-13 11:45 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-13 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 13:38 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-18 21:26 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 21:54 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 04:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-19 09:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 22:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-20 00:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-20 09:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
@ 2026-02-20 18:15 ` Nathan Bossart <nathandbossart@gmail.com>
2026-02-23 09:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
0 siblings, 1 reply; 178+ messages in thread
From: Nathan Bossart @ 2026-02-20 18:15 UTC (permalink / raw)
To: Nazir Bilal Yavuz <byavuz81@gmail.com>; +Cc: Manni Wood <manni.wood@enterprisedb.com>; KAZAR Ayoub <ma_kazar@esi.dz>; Neil Conway <neil.conway@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
On Fri, Feb 20, 2026 at 12:50:35PM +0300, Nazir Bilal Yavuz wrote:
> On Fri, 20 Feb 2026 at 03:09, Manni Wood <manni.wood@enterprisedb.com> wrote:
>> Nazir, I ran my benchmarks with more rows this time --- as many rows as
>> would fit on my test computers without exhausting their RAM disks. That
>> seems to have brought things more into line with what Ayoub saw. I did
>> get some small regressions, but I suspect those are not a big deal. (For
>> instance, on both machines I also noticed the occasional "truncate
>> table" would take longer than the others, despite my scripts' best
>> efforts to steady a CPU core and pin postmaster and children to that
>> core.)
Yeah, the couple of small regressions seem close to (or below) the noise
level, and IIUC yours were the only benchmarks that showed them, anyway.
Plus, I think we'll need this change regardless as a prerequisite for the
SIMD work.
> Thank you both for the benchmarks. Results look good to me!
Committed that part.
--
nathan
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 22:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-10 11:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-21 20:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Neil Conway <neil.conway@gmail.com>
2026-01-31 16:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 13:51 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 18:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 21:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-06 22:19 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 22:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-11 13:27 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-11 22:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-13 11:45 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-13 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 13:38 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-18 21:26 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 21:54 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 04:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-19 09:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 22:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-20 00:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-20 09:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-20 18:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
@ 2026-02-23 09:10 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 04:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
0 siblings, 1 reply; 178+ messages in thread
From: Nazir Bilal Yavuz @ 2026-02-23 09:10 UTC (permalink / raw)
To: Nathan Bossart <nathandbossart@gmail.com>; +Cc: Manni Wood <manni.wood@enterprisedb.com>; KAZAR Ayoub <ma_kazar@esi.dz>; Neil Conway <neil.conway@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
Hi,
On Fri, 20 Feb 2026 at 21:15, Nathan Bossart <nathandbossart@gmail.com> wrote:
>
> Yeah, the couple of small regressions seem close to (or below) the noise
> level, and IIUC yours were the only benchmarks that showed them, anyway.
> Plus, I think we'll need this change regardless as a prerequisite for the
> SIMD work.
>
> > Thank you both for the benchmarks. Results look good to me!
>
> Committed that part.
Thank you! Attaching the SIMD patch only.
--
Regards,
Nazir Bilal Yavuz
Microsoft
Attachments:
[text/x-patch] v10-0001-Speed-up-COPY-FROM-text-CSV-parsing-using-SIMD.patch (7.7K, ../../CAN55FZ3cBN_TncLVWyXAKm-KfewguN1AUjyRhoR6zL_QCxHh7A@mail.gmail.com/2-v10-0001-Speed-up-COPY-FROM-text-CSV-parsing-using-SIMD.patch)
download | inline diff:
From 9ef4e1376657b577cd4b4c42fb6a592ebd5fae24 Mon Sep 17 00:00:00 2001
From: Nazir Bilal Yavuz <byavuz81@gmail.com>
Date: Fri, 13 Feb 2026 13:28:55 +0300
Subject: [PATCH v10] Speed up COPY FROM text/CSV parsing using SIMD
This patch disables SIMD when SIMD encounters a special character which
is neither EOF nor EOL.
Author: Shinya Kato <shinya11.kato@gmail.com>
Author: Nazir Bilal Yavuz <byavuz81@gmail.com>
Reviewed-by: Kazar Ayoub <ma_kazar@esi.dz>
Reviewed-by: Nathan Bossart <nathandbossart@gmail.com>
Reviewed-by: Neil Conway <neil.conway@gmail.com>
Reviewed-by: Andrew Dunstan <andrew@dunslane.net>
Reviewed-by: Manni Wood <manni.wood@enterprisedb.com>
Reviewed-by: Mark Wong <markwkm@gmail.com>
Discussion: https://postgr.es/m/CAOzEurSW8cNr6TPKsjrstnPfhf4QyQqB4tnPXGGe8N4e_v7Jig%40mail.gmail.com
---
src/backend/commands/copyfrom.c | 3 +
src/backend/commands/copyfromparse.c | 135 ++++++++++++++++++++++-
src/include/commands/copyfrom_internal.h | 3 +
3 files changed, 137 insertions(+), 4 deletions(-)
diff --git a/src/backend/commands/copyfrom.c b/src/backend/commands/copyfrom.c
index 2b7556b287c..3dd159f15b2 100644
--- a/src/backend/commands/copyfrom.c
+++ b/src/backend/commands/copyfrom.c
@@ -1717,6 +1717,9 @@ BeginCopyFrom(ParseState *pstate,
cstate->cur_attval = NULL;
cstate->relname_only = false;
+ /* Initialize SIMD */
+ cstate->simd_enabled = true;
+
/*
* Allocate buffers for the input pipeline.
*
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index 6b00d49c50f..7bdf5681628 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -72,6 +72,7 @@
#include "miscadmin.h"
#include "pgstat.h"
#include "port/pg_bswap.h"
+#include "port/simd.h"
#include "utils/builtins.h"
#include "utils/rel.h"
@@ -142,7 +143,8 @@ static const char BinarySignature[11] = "PGCOPY\n\377\r\n\0";
/* non-export function prototypes */
static bool CopyReadLine(CopyFromState cstate, bool is_csv);
static pg_attribute_always_inline bool CopyReadLineText(CopyFromState cstate,
- bool is_csv);
+ bool is_csv,
+ bool simd_enabled);
static int CopyReadAttributesText(CopyFromState cstate);
static int CopyReadAttributesCSV(CopyFromState cstate);
static Datum CopyReadBinaryAttribute(CopyFromState cstate, FmgrInfo *flinfo,
@@ -1182,9 +1184,19 @@ CopyReadLine(CopyFromState cstate, bool is_csv)
* specialized code with fewer branches.
*/
if (is_csv)
- result = CopyReadLineText(cstate, true);
+ {
+ if (cstate->simd_enabled)
+ result = CopyReadLineText(cstate, true, true);
+ else
+ result = CopyReadLineText(cstate, true, false);
+ }
else
- result = CopyReadLineText(cstate, false);
+ {
+ if (cstate->simd_enabled)
+ result = CopyReadLineText(cstate, false, true);
+ else
+ result = CopyReadLineText(cstate, false, false);
+ }
if (result)
{
@@ -1252,7 +1264,7 @@ CopyReadLine(CopyFromState cstate, bool is_csv)
* CopyReadLineText - inner loop of CopyReadLine for text mode
*/
static pg_attribute_always_inline bool
-CopyReadLineText(CopyFromState cstate, bool is_csv)
+CopyReadLineText(CopyFromState cstate, bool is_csv, bool simd_enabled)
{
char *copy_input_buf;
int input_buf_ptr;
@@ -1267,6 +1279,14 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
char quotec = '\0';
char escapec = '\0';
+#ifndef USE_NO_SIMD
+ Vector8 nl = vector8_broadcast('\n');
+ Vector8 cr = vector8_broadcast('\r');
+ Vector8 bs = vector8_broadcast('\\');
+ Vector8 quote = vector8_broadcast(0);
+ Vector8 escape = vector8_broadcast(0);
+#endif
+
if (is_csv)
{
quotec = cstate->opts.quote[0];
@@ -1274,6 +1294,12 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
/* ignore special escape processing if it's the same as quotec */
if (quotec == escapec)
escapec = '\0';
+
+#ifndef USE_NO_SIMD
+ quote = vector8_broadcast(quotec);
+ if (quotec != escapec)
+ escape = vector8_broadcast(escapec);
+#endif
}
/*
@@ -1340,6 +1366,107 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
need_data = false;
}
+#ifndef USE_NO_SIMD
+
+ /*
+ * Use SIMD instructions to efficiently scan the input buffer for
+ * special characters (e.g., newline, carriage return, quote, and
+ * escape). This is faster than byte-by-byte iteration, especially on
+ * large buffers.
+ *
+ * We do not apply the SIMD fast path in either of the following
+ * cases: - When the previously processed character was an escape
+ * character (last_was_esc), since the next byte must be examined
+ * sequentially. - When the remaining buffer is smaller than one
+ * vector width (sizeof(Vector8)), since SIMD operates on fixed-size
+ * chunks.
+ *
+ * Note that, SIMD may become slower when the input contains many
+ * special characters. To avoid this regression, we disable SIMD for
+ * the rest of the input once we encounter a special character which
+ * is neither EOF nor EOL.
+ */
+ if (simd_enabled && !last_was_esc && copy_buf_len - input_buf_ptr > sizeof(Vector8))
+ {
+ Vector8 chunk;
+ Vector8 match = vector8_broadcast(0);
+
+ /* Load a chunk of data into a vector register */
+ vector8_load(&chunk, (const uint8 *) ©_input_buf[input_buf_ptr]);
+
+ if (is_csv)
+ {
+ /* \n and \r are not special inside quotes */
+ if (!in_quote)
+ match = vector8_or(vector8_eq(chunk, nl), vector8_eq(chunk, cr));
+
+ match = vector8_or(match, vector8_eq(chunk, quote));
+ if (escapec != '\0')
+ match = vector8_or(match, vector8_eq(chunk, escape));
+ }
+ else
+ {
+ match = vector8_or(vector8_eq(chunk, nl), vector8_eq(chunk, cr));
+ match = vector8_or(match, vector8_eq(chunk, bs));
+ }
+
+ /* Check if we found any special characters */
+ if (vector8_is_highbit_set(match))
+ {
+ /*
+ * Found a special character. Advance up to that point and let
+ * the scalar code handle it.
+ */
+ uint32 mask;
+ int advance;
+ char c1,
+ c2;
+ bool simd_hit_eol,
+ simd_hit_eof;
+
+ mask = vector8_highbit_mask(match);
+ advance = pg_rightmost_one_pos32(mask);
+
+ input_buf_ptr += advance;
+ c1 = copy_input_buf[input_buf_ptr];
+
+ /*
+ * Since we stopped within the chunk and ((copy_buf_len -
+ * input_buf_ptr) > sizeof(Vector8)) is true,
+ * copy_input_buf[input_buf_ptr + 1] is guaranteed to be
+ * readable.
+ */
+ c2 = copy_input_buf[input_buf_ptr + 1];
+
+ simd_hit_eof = (c1 == '\\' && c2 == '.' && !is_csv);
+ simd_hit_eol = (c1 == '\r' || c1 == '\n');
+
+ /*
+ * If (is_csv && in_quote), we shouldn't have picked up '\r'
+ * or '\n' in the first place.
+ */
+ Assert(!simd_hit_eol || !(is_csv && in_quote));
+
+ /*
+ * Do not disable SIMD when we hit EOL or EOF characters. In
+ * practice, it does not matter for EOF because parsing ends
+ * there, but we keep the behavior consistent.
+ */
+ if (!(simd_hit_eof || simd_hit_eol))
+ {
+ simd_enabled = false;
+ cstate->simd_enabled = false;
+ }
+ }
+ else
+ {
+ /* No special characters found, so skip the entire chunk */
+ input_buf_ptr += sizeof(Vector8);
+ continue;
+ }
+ }
+#endif
+
/* OK to fetch a character */
prev_raw_ptr = input_buf_ptr;
c = copy_input_buf[input_buf_ptr++];
diff --git a/src/include/commands/copyfrom_internal.h b/src/include/commands/copyfrom_internal.h
index 822ef33cf69..73ce777c52b 100644
--- a/src/include/commands/copyfrom_internal.h
+++ b/src/include/commands/copyfrom_internal.h
@@ -89,6 +89,9 @@ typedef struct CopyFromStateData
const char *cur_attval; /* current att value for error messages */
bool relname_only; /* don't output line number, att, etc. */
+ /* SIMD variables */
+ bool simd_enabled;
+
/*
* Working state
*/
--
2.47.3
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 22:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-10 11:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-21 20:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Neil Conway <neil.conway@gmail.com>
2026-01-31 16:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 13:51 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 18:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 21:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-06 22:19 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 22:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-11 13:27 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-11 22:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-13 11:45 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-13 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 13:38 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-18 21:26 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 21:54 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 04:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-19 09:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 22:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-20 00:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-20 09:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-20 18:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-23 09:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
@ 2026-02-24 04:44 ` Manni Wood <manni.wood@enterprisedb.com>
2026-02-24 13:57 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
0 siblings, 1 reply; 178+ messages in thread
From: Manni Wood @ 2026-02-24 04:44 UTC (permalink / raw)
To: Nazir Bilal Yavuz <byavuz81@gmail.com>; +Cc: Nathan Bossart <nathandbossart@gmail.com>; KAZAR Ayoub <ma_kazar@esi.dz>; Neil Conway <neil.conway@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
On Mon, Feb 23, 2026 at 3:10 AM Nazir Bilal Yavuz <byavuz81@gmail.com>
wrote:
> Hi,
>
> On Fri, 20 Feb 2026 at 21:15, Nathan Bossart <nathandbossart@gmail.com>
> wrote:
> >
> > Yeah, the couple of small regressions seem close to (or below) the noise
> > level, and IIUC yours were the only benchmarks that showed them, anyway.
> > Plus, I think we'll need this change regardless as a prerequisite for the
> > SIMD work.
> >
> > > Thank you both for the benchmarks. Results look good to me!
> >
> > Committed that part.
>
> Thank you! Attaching the SIMD patch only.
>
> --
> Regards,
> Nazir Bilal Yavuz
> Microsoft
>
Hello!
I ran some speed tests on Nazir's v10 SIMD-only patch. I'm a bit surprised
at the regression for x86 with wide rows for the 1/3rd special characters
scenarios. I'm hoping it's something I did wrong. If anyone else has
numbers to share, that would be excellent.
x86 NARROW master 50,000,000 rows
TXT : 26359.319000 ms
CSV : 25661.199750 ms
TXT with 1/3 escapes: 28170.085250 ms
CSV with 1/3 quotes: 32638.147500 ms
x86 NARROW v10 50,000,000 rows
TXT : 26416.331500 ms -0.216290% regression
CSV : 25318.727500 ms 1.334592% improvement
TXT with 1/3 escapes: 28608.007500 ms -1.554565% regression
CSV with 1/3 quotes: 32805.627750 ms -0.513143% regression
x86 WIDE master 500,000 rows
TXT : 26475.164250 ms
CSV : 31963.478500 ms
TXT with 1/3 escapes: 29671.120750 ms
CSV with 1/3 quotes: 40391.616250 ms
x86 WIDE v10 500,000 rows
TXT : 23067.046750 ms 12.872885% improvement
CSV : 23259.092250 ms 27.232287% improvement
TXT with 1/3 escapes: 31796.098250 ms -7.161770% regression
CSV with 1/3 quotes: 42925.792250 ms -6.274015% regression
arm NARROW master 25,000,000 rows
TXT : 10077.096250 ms
CSV : 10310.671250 ms
TXT with 1/3 escapes: 9893.155000 ms
CSV with 1/3 quotes: 12133.064750 ms
arm NARROW v10 25,000,000 rows
TXT : 10467.816750 ms -3.877312% regression
CSV : 9986.288000 ms 3.146092% improvement
TXT with 1/3 escapes: 10323.173750 ms -4.346629% regression
CSV with 1/3 quotes: 11843.611750 ms 2.385654% improvement
arm WIDE master 250,000 rows
TXT : 10568.344750 ms
CSV : 13046.610500 ms
TXT with 1/3 escapes: 12193.088500 ms
CSV with 1/3 quotes: 16629.319000 ms
arm WIDE v10 250,000 rows
TXT : 9064.959000 ms 14.225366% improvement
CSV : 9019.553250 ms 30.866693% improvement
TXT with 1/3 escapes: 12344.497250 ms -1.241759% regression
CSV with 1/3 quotes: 15495.863750 ms 6.816005% improvement
--
-- Manni Wood EDB: https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 22:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-10 11:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-21 20:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Neil Conway <neil.conway@gmail.com>
2026-01-31 16:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 13:51 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 18:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 21:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-06 22:19 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 22:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-11 13:27 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-11 22:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-13 11:45 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-13 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 13:38 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-18 21:26 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 21:54 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 04:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-19 09:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 22:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-20 00:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-20 09:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-20 18:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-23 09:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 04:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
@ 2026-02-24 13:57 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 15:07 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-24 17:48 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
0 siblings, 2 replies; 178+ messages in thread
From: Nazir Bilal Yavuz @ 2026-02-24 13:57 UTC (permalink / raw)
To: Manni Wood <manni.wood@enterprisedb.com>; +Cc: Nathan Bossart <nathandbossart@gmail.com>; KAZAR Ayoub <ma_kazar@esi.dz>; Neil Conway <neil.conway@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
Hi,
On Tue, 24 Feb 2026 at 07:44, Manni Wood <manni.wood@enterprisedb.com> wrote:
>
> Hello!
>
> I ran some speed tests on Nazir's v10 SIMD-only patch. I'm a bit surprised at the regression for x86 with wide rows for the 1/3rd special characters scenarios. I'm hoping it's something I did wrong. If anyone else has numbers to share, that would be excellent.
Thank you for doing this!
I see similar regression on the wide & CSV 1/3 case by using your
benchmark script. I didn't see this regression when I used my
benchmark while sharing v9 [1].
+-------------+---------------------------+---------------------------+
| | Text | CSV |
+-------------+-------------+-------------+-------------+-------------+
| WIDE TEST | None | 1/3 | None | 1/3 |
+-------------+-------------+-------------+-------------+-------------+
| Master | 9996 | 10769 | 11548 | 13960 |
+-------------+-------------+-------------+-------------+-------------+
| v10 | 8912 %-10.8 | 10902 %+1.2 | 8952 %-22.4 | 15123 %+8.3 |
+-------------+-------------+-------------+-------------+-------------+
| | | | | |
+-------------+-------------+-------------+-------------+-------------+
| | Text | | CSV |
+-------------+-------------+-------------+-------------+-------------+
| NARROW TEST | None | 1/3 | None | 1/3 |
+-------------+-------------+-------------+-------------+-------------+
| Master | 9441 | 9561 | 9734 | 9830 |
+-------------+-------------+-------------+-------------+-------------+
| v10 | 9291 %-1.5 | 9504 -%0.5 | 9644 %-0.9 | 10078 %-2.4 |
+-------------+-------------+-------------+-------------+-------------+
I will investigate this. However, please note that the current master
includes the inlining commit (dc592a4155), which makes the COPY FROM
faster. In my case,
1: current master without dc592a4155: 14400ms
2: current master: 13960ms (%3 improvement against #1)
3: current master + SIMD: 15123ms (%5 regression against #1 and %8
regression against #2)
Is it possible for you to do a similar test? I mean dropping
dc592a4155 from the current master and re-running the benchmark, that
would be helpful.
[1] https://postgr.es/m/CAN55FZ0MiFCgK26gRgE05a%3D_ggenkxDM8H%3DA2uTHpywczqt%3D-Q%40mail.gmail.com
--
Regards,
Nazir Bilal Yavuz
Microsoft
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 22:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-10 11:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-21 20:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Neil Conway <neil.conway@gmail.com>
2026-01-31 16:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 13:51 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 18:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 21:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-06 22:19 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 22:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-11 13:27 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-11 22:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-13 11:45 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-13 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 13:38 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-18 21:26 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 21:54 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 04:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-19 09:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 22:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-20 00:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-20 09:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-20 18:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-23 09:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 04:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-24 13:57 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
@ 2026-02-24 15:07 ` KAZAR Ayoub <ma_kazar@esi.dz>
1 sibling, 0 replies; 178+ messages in thread
From: KAZAR Ayoub @ 2026-02-24 15:07 UTC (permalink / raw)
To: Nazir Bilal Yavuz <byavuz81@gmail.com>; +Cc: Manni Wood <manni.wood@enterprisedb.com>; Nathan Bossart <nathandbossart@gmail.com>; Neil Conway <neil.conway@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
Hello,
On Tue, Feb 24, 2026 at 2:57 PM Nazir Bilal Yavuz <byavuz81@gmail.com>
wrote:
> Hi,
>
> On Tue, 24 Feb 2026 at 07:44, Manni Wood <manni.wood@enterprisedb.com>
> wrote:
> >
> > Hello!
> >
> > I ran some speed tests on Nazir's v10 SIMD-only patch. I'm a bit
> surprised at the regression for x86 with wide rows for the 1/3rd special
> characters scenarios. I'm hoping it's something I did wrong. If anyone else
> has numbers to share, that would be excellent.
>
> Thank you for doing this!
>
> I see similar regression on the wide & CSV 1/3 case by using your
> benchmark script. I didn't see this regression when I used my
> benchmark while sharing v9 [1].
>
> +-------------+---------------------------+---------------------------+
> | | Text | CSV |
> +-------------+-------------+-------------+-------------+-------------+
> | WIDE TEST | None | 1/3 | None | 1/3 |
> +-------------+-------------+-------------+-------------+-------------+
> | Master | 9996 | 10769 | 11548 | 13960 |
> +-------------+-------------+-------------+-------------+-------------+
> | v10 | 8912 %-10.8 | 10902 %+1.2 | 8952 %-22.4 | 15123 %+8.3 |
> +-------------+-------------+-------------+-------------+-------------+
> | | | | | |
> +-------------+-------------+-------------+-------------+-------------+
> | | Text | | CSV |
> +-------------+-------------+-------------+-------------+-------------+
> | NARROW TEST | None | 1/3 | None | 1/3 |
> +-------------+-------------+-------------+-------------+-------------+
> | Master | 9441 | 9561 | 9734 | 9830 |
> +-------------+-------------+-------------+-------------+-------------+
> | v10 | 9291 %-1.5 | 9504 -%0.5 | 9644 %-0.9 | 10078 %-2.4 |
> +-------------+-------------+-------------+-------------+-------------+
>
> I will investigate this. However, please note that the current master
> includes the inlining commit (dc592a4155), which makes the COPY FROM
> faster. In my case,
>
> 1: current master without dc592a4155: 14400ms
> 2: current master: 13960ms (%3 improvement against #1)
> 3: current master + SIMD: 15123ms (%5 regression against #1 and %8
> regression against #2)
>
> Is it possible for you to do a similar test? I mean dropping
> dc592a4155 from the current master and re-running the benchmark, that
> would be helpful.
>
> [1]
> https://postgr.es/m/CAN55FZ0MiFCgK26gRgE05a%3D_ggenkxDM8H%3DA2uTHpywczqt%3D-Q%40mail.gmail.com
Here are some numbers for v10 from my end, these are multiple long runs:
Master contains the previous inlining patch.
This is on an Intel I7-1255U CPU
WIDE (500k rows)
TXT | none
Master avg: 20,721 ms
New avg: 17,980 ms
Improvement: -13.23%
CSV | none
Master avg: 26,608 ms
New avg: 18,433 ms
Improvement: -30.73%
TXT | escape
Master avg: 25,069 ms
New avg: 22,910 ms
Improvement: -8.61%
CSV | quote
Master avg: 31,931 ms
New avg: 31,493 ms
Improvement: -1.37%
--------------------------------------
NARROW (15M rows)
TXT | none
Master avg: 20,687 ms
New avg: 20,824 ms
Regression: +0.67%
CSV | none
Master avg: 21,187 ms
New avg: 21,153 ms
Improvement: -0.16%
TXT | escape
Master avg: 20,870 ms
New avg: 21,341 ms
Regression: +2.25%
CSV | quote
Master avg: 22,074 ms
New avg: 22,267 ms
Regression: +0.87%
For narrow that would be mostly noise and extra branch effects.
Regards,
Ayoub
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 22:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-10 11:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-21 20:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Neil Conway <neil.conway@gmail.com>
2026-01-31 16:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 13:51 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 18:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 21:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-06 22:19 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 22:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-11 13:27 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-11 22:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-13 11:45 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-13 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 13:38 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-18 21:26 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 21:54 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 04:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-19 09:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 22:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-20 00:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-20 09:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-20 18:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-23 09:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 04:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-24 13:57 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
@ 2026-02-24 17:48 ` Nathan Bossart <nathandbossart@gmail.com>
2026-02-25 04:06 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-25 14:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
1 sibling, 2 replies; 178+ messages in thread
From: Nathan Bossart @ 2026-02-24 17:48 UTC (permalink / raw)
To: Nazir Bilal Yavuz <byavuz81@gmail.com>; +Cc: Manni Wood <manni.wood@enterprisedb.com>; KAZAR Ayoub <ma_kazar@esi.dz>; Neil Conway <neil.conway@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
On Tue, Feb 24, 2026 at 04:57:21PM +0300, Nazir Bilal Yavuz wrote:
> I will investigate this. However, please note that the current master
> includes the inlining commit (dc592a4155), which makes the COPY FROM
> faster. In my case,
>
> 1: current master without dc592a4155: 14400ms
> 2: current master: 13960ms (%3 improvement against #1)
> 3: current master + SIMD: 15123ms (%5 regression against #1 and %8
> regression against #2)
>
> Is it possible for you to do a similar test? I mean dropping
> dc592a4155 from the current master and re-running the benchmark, that
> would be helpful.
IMHO as long as the difference from v18 looks reasonable, commit-by-commit
regressions and improvements that even out in the end are okay. That's
perhaps a bit of mental gymnastics (e.g., what if we had committed the
inlining patch for v18?), but I believe that's how we've dealt with similar
problems in the past. But maybe there are ways to avoid even these
in-development regressions, too...
--
nathan
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 22:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-10 11:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-21 20:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Neil Conway <neil.conway@gmail.com>
2026-01-31 16:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 13:51 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 18:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 21:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-06 22:19 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 22:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-11 13:27 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-11 22:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-13 11:45 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-13 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 13:38 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-18 21:26 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 21:54 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 04:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-19 09:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 22:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-20 00:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-20 09:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-20 18:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-23 09:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 04:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-24 13:57 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 17:48 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
@ 2026-02-25 04:06 ` Manni Wood <manni.wood@enterprisedb.com>
1 sibling, 0 replies; 178+ messages in thread
From: Manni Wood @ 2026-02-25 04:06 UTC (permalink / raw)
To: Nathan Bossart <nathandbossart@gmail.com>; +Cc: Nazir Bilal Yavuz <byavuz81@gmail.com>; KAZAR Ayoub <ma_kazar@esi.dz>; Neil Conway <neil.conway@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
On Tue, Feb 24, 2026 at 11:48 AM Nathan Bossart <nathandbossart@gmail.com>
wrote:
> On Tue, Feb 24, 2026 at 04:57:21PM +0300, Nazir Bilal Yavuz wrote:
> > I will investigate this. However, please note that the current master
> > includes the inlining commit (dc592a4155), which makes the COPY FROM
> > faster. In my case,
> >
> > 1: current master without dc592a4155: 14400ms
> > 2: current master: 13960ms (%3 improvement against #1)
> > 3: current master + SIMD: 15123ms (%5 regression against #1 and %8
> > regression against #2)
> >
> > Is it possible for you to do a similar test? I mean dropping
> > dc592a4155 from the current master and re-running the benchmark, that
> > would be helpful.
>
> IMHO as long as the difference from v18 looks reasonable, commit-by-commit
> regressions and improvements that even out in the end are okay. That's
> perhaps a bit of mental gymnastics (e.g., what if we had committed the
> inlining patch for v18?), but I believe that's how we've dealt with similar
> problems in the past. But maybe there are ways to avoid even these
> in-development regressions, too...
>
> --
> nathan
>
Oh yes, I see now.
Commit 18bcdb75 is just before the v9 patch got applied, so I used that as
"old master" and compared that with master (v9 applied) and then "master
(v9 applied) + v10 applied".
arm NARROW old master 18bcdb75
TXT : 10997.568250 ms
CSV : 10797.549000 ms
TXT with 1/3 escapes: 10299.047000 ms
CSV with 1/3 quotes: 12559.385750 ms
arm NARROW master (v9 applied)
TXT : 10077.096250 ms 8.369778% improvement
CSV : 10310.671250 ms 4.509151% improvement
TXT with 1/3 escapes: 9893.155000 ms 3.941064% improvement
CSV with 1/3 quotes: 12133.064750 ms 3.394441% improvement
arm NARROW v10
TXT : 10467.816750 ms 4.816988% improvement
CSV : 9986.288000 ms 7.513381% improvement
TXT with 1/3 escapes: 10323.173750 ms -0.234262% regression
CSV with 1/3 quotes: 11843.611750 ms 5.699116% improvement
arm WIDE old master 18bcdb75
TXT : 11825.771250 ms
CSV : 13907.074000 ms
TXT with 1/3 escapes: 13430.691250 ms
CSV with 1/3 quotes: 17557.954500 ms
arm WIDE master (v9 applied)
TXT : 10568.344750 ms 10.632934% improvement
CSV : 13046.610500 ms 6.187236% improvement
TXT with 1/3 escapes: 12193.088500 ms 9.214736% improvement
CSV with 1/3 quotes: 16629.319000 ms 5.288973% improvement
arm WIDE v10
TXT : 9064.959000 ms 23.345727% improvement
CSV : 9019.553250 ms 35.144134% improvement
TXT with 1/3 escapes: 12344.497250 ms 8.087402% improvement
CSV with 1/3 quotes: 15495.863750 ms 11.744482% improvement
x86 NARROW old master 18bcdb75
TXT : 25909.060500 ms
CSV : 28137.591250 ms
TXT with 1/3 escapes: 27794.177000 ms
CSV with 1/3 quotes: 34541.704750 ms
x86 NARROW master
TXT : 26359.319000 ms -1.737842% regression
CSV : 25661.199750 ms 8.801007% improvement
TXT with 1/3 escapes: 28170.085250 ms -1.352471% regression
CSV with 1/3 quotes: 32638.147500 ms 5.510895% improvement
x86 NARROW v10
TXT : 26416.331500 ms -1.957890% regression
CSV : 25318.727500 ms 10.018142% improvement
TXT with 1/3 escapes: 28608.007500 ms -2.928061% regression
CSV with 1/3 quotes: 32805.627750 ms 5.026032% improvement
x86 WIDE old master 18bcdb75
TXT : 28778.426500 ms
CSV : 35671.908000 ms
TXT with 1/3 escapes: 32441.549750 ms
CSV with 1/3 quotes: 47024.416000 ms
x86 WIDE master
TXT : 26475.164250 ms 8.003434% improvement
CSV : 31963.478500 ms 10.395938% improvement
TXT with 1/3 escapes: 29671.120750 ms 8.539755% improvement
CSV with 1/3 quotes: 40391.616250 ms 14.105012% improvement
x86 WIDE v10
TXT : 23067.046750 ms 19.846046% improvement
CSV : 23259.092250 ms 34.797174% improvement
TXT with 1/3 escapes: 31796.098250 ms 1.989583% improvement
CSV with 1/3 quotes: 42925.792250 ms 8.715948% improvement
--
-- Manni Wood EDB: https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 22:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-10 11:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-21 20:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Neil Conway <neil.conway@gmail.com>
2026-01-31 16:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 13:51 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 18:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 21:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-06 22:19 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 22:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-11 13:27 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-11 22:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-13 11:45 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-13 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 13:38 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-18 21:26 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 21:54 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 04:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-19 09:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 22:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-20 00:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-20 09:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-20 18:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-23 09:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 04:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-24 13:57 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 17:48 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
@ 2026-02-25 14:24 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-26 12:19 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-02 19:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
1 sibling, 2 replies; 178+ messages in thread
From: Nazir Bilal Yavuz @ 2026-02-25 14:24 UTC (permalink / raw)
To: Nathan Bossart <nathandbossart@gmail.com>; +Cc: Manni Wood <manni.wood@enterprisedb.com>; KAZAR Ayoub <ma_kazar@esi.dz>; Neil Conway <neil.conway@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
Hi,
On Tue, 24 Feb 2026 at 20:48, Nathan Bossart <nathandbossart@gmail.com> wrote:
>
> On Tue, Feb 24, 2026 at 04:57:21PM +0300, Nazir Bilal Yavuz wrote:
> > I will investigate this. However, please note that the current master
> > includes the inlining commit (dc592a4155), which makes the COPY FROM
> > faster. In my case,
> >
> > 1: current master without dc592a4155: 14400ms
> > 2: current master: 13960ms (%3 improvement against #1)
> > 3: current master + SIMD: 15123ms (%5 regression against #1 and %8
> > regression against #2)
> >
> > Is it possible for you to do a similar test? I mean dropping
> > dc592a4155 from the current master and re-running the benchmark, that
> > would be helpful.
>
> IMHO as long as the difference from v18 looks reasonable, commit-by-commit
> regressions and improvements that even out in the end are okay. That's
> perhaps a bit of mental gymnastics (e.g., what if we had committed the
> inlining patch for v18?), but I believe that's how we've dealt with similar
> problems in the past. But maybe there are ways to avoid even these
> in-development regressions, too...
I agree with you. However, unfortunately, I see regression on master +
v10 compared to REL_18_3 (62d6c7d3df6).
Thank you Kazar and Manni for benchmarks in [1] and [2]!
I am still able to reproduce regression for the 'wide & CSV 1/3' case
[3] by using Manni's benchmark script. I constantly see ~%5
regression, I am just curious if I am doing something wrong. I am a
bit surprised because I didn't see this regression before, also Kazar
and Manni don't see any regression in their [1] and [2] benchmarks. I
am still investigating this regression. Hopefully, I will come back
with more information soon.
If anyone has any suggestions/ideas, please let me know!
[1] https://postgr.es/m/CA%2BK2RukFH57QPAfTEzvy7PEyrLzav3HkyCiu-2yqR%2BuW_Niorw%40mail.gmail.com
[2] https://postgr.es/m/CAKWEB6oT5KbyF%2BuRRhjjJi7p2PmRdOzxp3T6vFcN04BCR-%3DB2w%40mail.gmail.com
[3]
1: current master without dc592a4155: 14400ms
2: current master: 13960ms (%3 improvement against #1)
3: current master + v10: 15123ms (%5 regression against #1 and %8
regression against #2)
--
Regards,
Nazir Bilal Yavuz
Microsoft
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 22:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-10 11:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-21 20:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Neil Conway <neil.conway@gmail.com>
2026-01-31 16:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 13:51 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 18:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 21:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-06 22:19 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 22:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-11 13:27 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-11 22:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-13 11:45 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-13 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 13:38 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-18 21:26 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 21:54 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 04:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-19 09:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 22:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-20 00:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-20 09:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-20 18:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-23 09:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 04:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-24 13:57 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 17:48 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-25 14:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
@ 2026-02-26 12:19 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-26 14:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
1 sibling, 1 reply; 178+ messages in thread
From: Nazir Bilal Yavuz @ 2026-02-26 12:19 UTC (permalink / raw)
To: Nathan Bossart <nathandbossart@gmail.com>; +Cc: Manni Wood <manni.wood@enterprisedb.com>; KAZAR Ayoub <ma_kazar@esi.dz>; Neil Conway <neil.conway@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
Hi,
On Wed, 25 Feb 2026 at 17:24, Nazir Bilal Yavuz <byavuz81@gmail.com> wrote:
>
> I agree with you. However, unfortunately, I see regression on master +
> v10 compared to REL_18_3 (62d6c7d3df6).
>
> Thank you Kazar and Manni for benchmarks in [1] and [2]!
Kazar and Manni, if possible could you please share the build commands
you use? I see regressions for an inlining patch (dc592a4155) too when
I build postgres with -O2.
My build commands are:
-O2: meson setup buildtype=debugoptimized ...
-O3: meson setup buildtype=release ...
This is a wide benchmark only, old master means b2ff2a0b529 without
0001 inlining patch (dc592a4155):
+--------------------------+---------------+---------------+
| | Text | CSV |
+--------------------------+-------+-------+-------+-------+
| -O2 | None | 1/3 | None | 1/3 |
+--------------------------+-------+-------+-------+-------+
| Old Master | 10440 | 11000 | 11940 | 13600 |
+--------------------------+-------+-------+-------+-------+
| Old Master + 0001 | 10140 | 10800 | 11600 | 14300 |
+--------------------------+-------+-------+-------+-------+
| Old Master + 0001 + 0002 | 9000 | 11000 | 8850 | 15300 |
+--------------------------+-------+-------+-------+-------+
| | | | | |
+--------------------------+-------+-------+-------+-------+
| | Text | | CSV |
+--------------------------+-------+-------+-------+-------+
| -O3 | None | 1/3 | None | 1/3 |
+--------------------------+-------+-------+-------+-------+
| Old Master | 10440 | 11200 | 12200 | 14390 |
+--------------------------+-------+-------+-------+-------+
| Old Master + 0001 | 10000 | 10700 | 11540 | 13960 |
+--------------------------+-------+-------+-------+-------+
| Old Master + 0001 + 0002 | 8880 | 10900 | 8900 | 15100 |
+--------------------------+-------+-------+-------+-------+
This result shows that when we compare v18 and v18 + SIMD (0001 +
0002), there is only regression for the CSV 1/3 case. The regression
is %12.5 for the -O2 and %5 for the -O3.
--
Regards,
Nazir Bilal Yavuz
Microsoft
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 22:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-10 11:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-21 20:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Neil Conway <neil.conway@gmail.com>
2026-01-31 16:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 13:51 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 18:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 21:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-06 22:19 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 22:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-11 13:27 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-11 22:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-13 11:45 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-13 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 13:38 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-18 21:26 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 21:54 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 04:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-19 09:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 22:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-20 00:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-20 09:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-20 18:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-23 09:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 04:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-24 13:57 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 17:48 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-25 14:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-26 12:19 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
@ 2026-02-26 14:31 ` KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-26 14:36 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
0 siblings, 1 reply; 178+ messages in thread
From: KAZAR Ayoub @ 2026-02-26 14:31 UTC (permalink / raw)
To: Nazir Bilal Yavuz <byavuz81@gmail.com>; +Cc: Nathan Bossart <nathandbossart@gmail.com>; Manni Wood <manni.wood@enterprisedb.com>; Neil Conway <neil.conway@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
Hello,
On Thu, Feb 26, 2026 at 1:19 PM Nazir Bilal Yavuz <byavuz81@gmail.com>
wrote:
> Hi,
>
> On Wed, 25 Feb 2026 at 17:24, Nazir Bilal Yavuz <byavuz81@gmail.com>
> wrote:
> >
> > I agree with you. However, unfortunately, I see regression on master +
> > v10 compared to REL_18_3 (62d6c7d3df6).
> >
> > Thank you Kazar and Manni for benchmarks in [1] and [2]!
>
> Kazar and Manni, if possible could you please share the build commands
> you use? I see regressions for an inlining patch (dc592a4155) too when
> I build postgres with -O2.
>
> My build commands are:
>
> -O2: meson setup buildtype=debugoptimized ...
>
> -O3: meson setup buildtype=release ...
All my builds are with CFLAGS='-O2 -g'
Regards,
Ayoub
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 22:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-10 11:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-21 20:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Neil Conway <neil.conway@gmail.com>
2026-01-31 16:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 13:51 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 18:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 21:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-06 22:19 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 22:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-11 13:27 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-11 22:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-13 11:45 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-13 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 13:38 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-18 21:26 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 21:54 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 04:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-19 09:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 22:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-20 00:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-20 09:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-20 18:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-23 09:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 04:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-24 13:57 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 17:48 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-25 14:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-26 12:19 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-26 14:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
@ 2026-02-26 14:36 ` Manni Wood <manni.wood@enterprisedb.com>
2026-02-26 15:32 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
0 siblings, 1 reply; 178+ messages in thread
From: Manni Wood @ 2026-02-26 14:36 UTC (permalink / raw)
To: KAZAR Ayoub <ma_kazar@esi.dz>; +Cc: Nazir Bilal Yavuz <byavuz81@gmail.com>; Nathan Bossart <nathandbossart@gmail.com>; Neil Conway <neil.conway@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
On Thu, Feb 26, 2026 at 8:31 AM KAZAR Ayoub <ma_kazar@esi.dz> wrote:
> Hello,
>
> On Thu, Feb 26, 2026 at 1:19 PM Nazir Bilal Yavuz <byavuz81@gmail.com>
> wrote:
>
>> Hi,
>>
>> On Wed, 25 Feb 2026 at 17:24, Nazir Bilal Yavuz <byavuz81@gmail.com>
>> wrote:
>> >
>> > I agree with you. However, unfortunately, I see regression on master +
>> > v10 compared to REL_18_3 (62d6c7d3df6).
>> >
>> > Thank you Kazar and Manni for benchmarks in [1] and [2]!
>>
>> Kazar and Manni, if possible could you please share the build commands
>> you use? I see regressions for an inlining patch (dc592a4155) too when
>> I build postgres with -O2.
>>
>> My build commands are:
>>
>> -O2: meson setup buildtype=debugoptimized ...
>>
>> -O3: meson setup buildtype=release ...
>
> All my builds are with CFLAGS='-O2 -g'
>
> Regards,
> Ayoub
>
Hello!
I have been building with this command:
meson setup build --prefix=/home/mwood/compiled-pg-instances/${BRANCH}
--buildtype=debugoptimized
And in my notes I have "If I use `--buildtype=debugoptimized` it optimizes
`-O2` and uses `-g`"
Best,
-Manni
--
-- Manni Wood EDB: https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 22:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-10 11:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-21 20:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Neil Conway <neil.conway@gmail.com>
2026-01-31 16:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 13:51 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 18:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 21:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-06 22:19 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 22:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-11 13:27 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-11 22:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-13 11:45 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-13 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 13:38 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-18 21:26 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 21:54 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 04:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-19 09:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 22:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-20 00:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-20 09:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-20 18:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-23 09:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 04:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-24 13:57 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 17:48 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-25 14:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-26 12:19 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-26 14:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-26 14:36 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
@ 2026-02-26 15:32 ` Manni Wood <manni.wood@enterprisedb.com>
2026-02-26 15:51 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
0 siblings, 1 reply; 178+ messages in thread
From: Manni Wood @ 2026-02-26 15:32 UTC (permalink / raw)
To: KAZAR Ayoub <ma_kazar@esi.dz>; +Cc: Nazir Bilal Yavuz <byavuz81@gmail.com>; Nathan Bossart <nathandbossart@gmail.com>; Neil Conway <neil.conway@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
I have a thought and a question:
My notes say "If I use `--buildtype=release` it optimizes `-O2` and the
executable contains no debug symbols."
So, seeing as end users will presumably be seeing the performance generated
by `--buildtype=release`, should we be building with that for all
performance testing?
Best,
-Manni
On Thu, Feb 26, 2026 at 8:36 AM Manni Wood <manni.wood@enterprisedb.com>
wrote:
>
>
> On Thu, Feb 26, 2026 at 8:31 AM KAZAR Ayoub <ma_kazar@esi.dz> wrote:
>
>> Hello,
>>
>> On Thu, Feb 26, 2026 at 1:19 PM Nazir Bilal Yavuz <byavuz81@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> On Wed, 25 Feb 2026 at 17:24, Nazir Bilal Yavuz <byavuz81@gmail.com>
>>> wrote:
>>> >
>>> > I agree with you. However, unfortunately, I see regression on master +
>>> > v10 compared to REL_18_3 (62d6c7d3df6).
>>> >
>>> > Thank you Kazar and Manni for benchmarks in [1] and [2]!
>>>
>>> Kazar and Manni, if possible could you please share the build commands
>>> you use? I see regressions for an inlining patch (dc592a4155) too when
>>> I build postgres with -O2.
>>>
>>> My build commands are:
>>>
>>> -O2: meson setup buildtype=debugoptimized ...
>>>
>>> -O3: meson setup buildtype=release ...
>>
>> All my builds are with CFLAGS='-O2 -g'
>>
>> Regards,
>> Ayoub
>>
>
> Hello!
>
> I have been building with this command:
>
> meson setup build --prefix=/home/mwood/compiled-pg-instances/${BRANCH}
> --buildtype=debugoptimized
>
> And in my notes I have "If I use `--buildtype=debugoptimized` it optimizes
> `-O2` and uses `-g`"
>
> Best,
> -Manni
> --
> -- Manni Wood EDB: https://www.enterprisedb.com
>
--
-- Manni Wood EDB: https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 22:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-10 11:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-21 20:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Neil Conway <neil.conway@gmail.com>
2026-01-31 16:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 13:51 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 18:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 21:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-06 22:19 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 22:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-11 13:27 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-11 22:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-13 11:45 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-13 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 13:38 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-18 21:26 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 21:54 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 04:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-19 09:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 22:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-20 00:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-20 09:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-20 18:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-23 09:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 04:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-24 13:57 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 17:48 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-25 14:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-26 12:19 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-26 14:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-26 14:36 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-26 15:32 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
@ 2026-02-26 15:51 ` KAZAR Ayoub <ma_kazar@esi.dz>
0 siblings, 0 replies; 178+ messages in thread
From: KAZAR Ayoub @ 2026-02-26 15:51 UTC (permalink / raw)
To: Manni Wood <manni.wood@enterprisedb.com>; +Cc: Nazir Bilal Yavuz <byavuz81@gmail.com>; Nathan Bossart <nathandbossart@gmail.com>; Neil Conway <neil.conway@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
On Thu, Feb 26, 2026 at 4:32 PM Manni Wood <manni.wood@enterprisedb.com>
wrote:
> I have a thought and a question:
>
> My notes say "If I use `--buildtype=release` it optimizes `-O2` and the
> executable contains no debug symbols."
>
That would be `debugoptimized` not `release`, from [1] i see that `release`
is -O3 with no debug.
>
> So, seeing as end users will presumably be seeing the performance
> generated by `--buildtype=release`, should we be building with that for all
> performance testing?
>
I know that Debian builds with 'CFLAGS=-g -O2 -flto=auto -ffat-lto-objects
-flto=auto -ffat-lto-objects -fstack-protector-strong -Wformat
-Werror=format-security -fno-omit-frame-pointer'
'LDFLAGS=-Wl,-Bsymbolic-functions -flto=auto -ffat-lto-objects -flto=auto
-Wl,-z,relro -Wl,-z,now' ; this is from pg_config for v18.
[1] https://mesonbuild.com/Builtin-options.html
Regards,
Ayoub
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 22:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-10 11:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-21 20:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Neil Conway <neil.conway@gmail.com>
2026-01-31 16:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 13:51 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 18:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 21:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-06 22:19 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 22:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-11 13:27 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-11 22:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-13 11:45 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-13 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 13:38 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-18 21:26 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 21:54 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 04:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-19 09:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 22:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-20 00:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-20 09:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-20 18:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-23 09:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 04:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-24 13:57 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 17:48 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-25 14:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
@ 2026-03-02 19:55 ` Nathan Bossart <nathandbossart@gmail.com>
2026-03-04 15:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
1 sibling, 1 reply; 178+ messages in thread
From: Nathan Bossart @ 2026-03-02 19:55 UTC (permalink / raw)
To: Nazir Bilal Yavuz <byavuz81@gmail.com>; +Cc: Manni Wood <manni.wood@enterprisedb.com>; KAZAR Ayoub <ma_kazar@esi.dz>; Neil Conway <neil.conway@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
On Wed, Feb 25, 2026 at 05:24:27PM +0300, Nazir Bilal Yavuz wrote:
> If anyone has any suggestions/ideas, please let me know!
A couple of random ideas:
* Additional inlining for callers. I looked around a little bit and didn't
see any great candidates, so I don't have much faith in this, but maybe
you'll see something I don't.
* Disable SIMD if we are consistently getting small rows. That won't help
your "wide & CSV 1/3" case in all likelihood, but perhaps it'll help with
the regression for narrow rows described elsewhere.
* Surround the variable initializations with "if (simd_enabled)".
Presumably compilers are smart enough to remove those in the non-SIMD paths
already, but it could be worth a try.
* Add simd_enabled function parameter to CopyReadLine(),
NextCopyFromRawFieldsInternal(), and CopyFromTextLikeOneRow(), and do the
bool literal trick in CopyFrom{Text,CSV}OneRow(). That could encourage the
compiler to do some additional optimizations to reduce branching.
--
nathan
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 22:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-10 11:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-21 20:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Neil Conway <neil.conway@gmail.com>
2026-01-31 16:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 13:51 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 18:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 21:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-06 22:19 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 22:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-11 13:27 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-11 22:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-13 11:45 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-13 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 13:38 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-18 21:26 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 21:54 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 04:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-19 09:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 22:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-20 00:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-20 09:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-20 18:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-23 09:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 04:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-24 13:57 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 17:48 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-25 14:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-02 19:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
@ 2026-03-04 15:15 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-05 21:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2026-03-09 18:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
0 siblings, 2 replies; 178+ messages in thread
From: Nazir Bilal Yavuz @ 2026-03-04 15:15 UTC (permalink / raw)
To: Nathan Bossart <nathandbossart@gmail.com>; +Cc: Manni Wood <manni.wood@enterprisedb.com>; KAZAR Ayoub <ma_kazar@esi.dz>; Neil Conway <neil.conway@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
Hi,
On Mon, 2 Mar 2026 at 22:55, Nathan Bossart <nathandbossart@gmail.com> wrote:
>
> On Wed, Feb 25, 2026 at 05:24:27PM +0300, Nazir Bilal Yavuz wrote:
> > If anyone has any suggestions/ideas, please let me know!
I am able to fix the problem. My first assumption was that the
branching of SIMD code caused that problem, so I moved SIMD code to
the CopyReadLineTextSIMDHelper() function. Then I moved this
CopyReadLineTextSIMDHelper() to top of CopyReadLineText(), by doing
that we won't have any branching in the non-SIMD (scalar) code path.
This didn't solve the problem and then I realized that even though I
disable SIMD code path with 'if (false)', there is still regression
but if I comment all of the 'if (cstate->simd_enabled)' branch, then
there is no regression at all.
To find out more, I compared assembly outputs of both and found out
the possible reason. What I understood is that the compiler can't
promote a variable to register, instead these variables live in the
stack; which is slower. Please see the two different assembly outputs:
Slow code:
c = copy_input_buf[input_buf_ptr++];
db0: 48 8b 55 b8 mov -0x48(%rbp),%rdx
db4: 48 63 c6 movslq %esi,%rax
db7: 44 8d 66 01 lea 0x1(%rsi),%r12d
dbb: 44 89 65 cc mov %r12d,-0x34(%rbp)
dbf: 0f be 14 02 movsbl (%rdx,%rax,1),%edx
Fast code:
c = copy_input_buf[input_buf_ptr++];
d80: 49 63 c4 movslq %r12d,%rax
d83: 45 8d 5c 24 01 lea 0x1(%r12),%r11d
d88: 41 0f be 04 06 movsbl (%r14,%rax,1),%eax
And the reason for that is sending the address of input_buf_ptr to a
CopyReadLineTextSIMDHelper(..., &input_buf_ptr). If I change it to
this:
int temp_input_buf_ptr = input_buf_ptr;
CopyReadLineTextSIMDHelper(..., &temp_input_buf_ptr);
Then there is no regression. However, I am still not completely sure
if that is the same problem in the v10, I am planning to spend more
time debugging this.
> A couple of random ideas:
>
> * Additional inlining for callers. I looked around a little bit and didn't
> see any great candidates, so I don't have much faith in this, but maybe
> you'll see something I don't.
I agree with you. CopyReadLineText() is already quite a big function.
> * Disable SIMD if we are consistently getting small rows. That won't help
> your "wide & CSV 1/3" case in all likelihood, but perhaps it'll help with
> the regression for narrow rows described elsewhere.
I implemented this, two consecutive small rows disables SIMD.
> * Surround the variable initializations with "if (simd_enabled)".
> Presumably compilers are smart enough to remove those in the non-SIMD paths
> already, but it could be worth a try.
Done.
> * Add simd_enabled function parameter to CopyReadLine(),
> NextCopyFromRawFieldsInternal(), and CopyFromTextLikeOneRow(), and do the
> bool literal trick in CopyFrom{Text,CSV}OneRow(). That could encourage the
> compiler to do some additional optimizations to reduce branching.
I think we don't need this. At least the implementation with
CopyReadLineTextSIMDHelper() doesn't need this since branching will be
at the top and it will be once per line.
I think v11 looks better compared to v10. I liked the
CopyReadLineTextSIMDHelper() helper function. I also liked it being at
the top of CopyReadLineText(), not being in the scalar path. This
gives us more optimization options without affecting the scalar path.
Here are the new benchmark results, I benchmarked the changes with
both -O2 and -O3 and also both with and without 'changing
default_toast_compression to lz4' commit (65def42b1d5). Benchmark
results show that there is no regression and the performance
improvement is much bigger with 65def42b1d5, it is close to 2x for
text format and more than 2x for the csv format.
------------------------------
Benchmark results:
With 65def42b1d5:
+---------------------------------------------------------+
| Optimization: -O2 |
+--------------------------+--------------+---------------+
| | Text | CSV |
+--------------------------+------+-------+-------+-------+
| WIDE | None | 1/3 | None | 1/3 |
+--------------------------+------+-------+-------+-------+
| Old Master | 4220 | 4780 | 5930 | 8250 |
+--------------------------+------+-------+-------+-------+
| Old Master + 0001 + 0002 | 2520 | 4500 | 2520 | 7800 |
+--------------------------+------+-------+-------+-------+
| | | | | |
+--------------------------+------+-------+-------+-------+
| | Text | CSV |
+--------------------------+------+-------+-------+-------+
| NARROW | None | 1/3 | None | 1/3 |
+--------------------------+------+-------+-------+-------+
| Old Master | 9920 | 10100 | 10200 | 10470 |
+--------------------------+------+-------+-------+-------+
| Old Master + 0001 + 0002 | 9970 | 10000 | 10180 | 10350 |
+--------------------------+------+-------+-------+-------+
| |
+---------------------------------------------------------+
| |
+---------------------------------------------------------+
| Optimization: -O3 |
+--------------------------+--------------+---------------+
| | Text | CSV |
+--------------------------+------+-------+-------+-------+
| WIDE | None | 1/3 | None | 1/3 |
+--------------------------+------+-------+-------+-------+
| Old Master | 4100 | 4900 | 6200 | 8300 |
+--------------------------+------+-------+-------+-------+
| Old Master + 0001 + 0002 | 2470 | 4440 | 2570 | 7700 |
+--------------------------+------+-------+-------+-------+
| | | | | |
+--------------------------+------+-------+-------+-------+
| | Text | CSV |
+--------------------------+------+-------+-------+-------+
| NARROW | None | 1/3 | None | 1/3 |
+--------------------------+------+-------+-------+-------+
| Old Master | 9530 | 9690 | 9800 | 10080 |
+--------------------------+------+-------+-------+-------+
| Old Master + 0001 + 0002 | 9350 | 9450 | 9700 | 10000 |
+--------------------------+------+-------+-------+-------+
------------------------------
Without 65def42b1d5:
+----------------------------------------------------------+
| Optimization: -O2 |
+--------------------------+---------------+---------------+
| | Text | CSV |
+--------------------------+-------+-------+-------+-------+
| WIDE | None | 1/3 | None | 1/3 |
+--------------------------+-------+-------+-------+-------+
| Old Master | 10550 | 11030 | 12250 | 14400 |
+--------------------------+-------+-------+-------+-------+
| Old Master + 0001 + 0002 | 8890 | 10700 | 8870 | 14070 |
+--------------------------+-------+-------+-------+-------+
| | | | | |
+--------------------------+-------+-------+-------+-------+
| | Text | CSV |
+--------------------------+-------+-------+-------+-------+
| NARROW | None | 1/3 | None | 1/3 |
+--------------------------+-------+-------+-------+-------+
| Old Master | 9921 | 10205 | 10123 | 10420 |
+--------------------------+-------+-------+-------+-------+
| Old Master + 0001 + 0002 | 9880 | 10070 | 10150 | 10400 |
+--------------------------+-------+-------+-------+-------+
| |
+----------------------------------------------------------+
| |
+----------------------------------------------------------+
| Optimization: -O3 |
+--------------------------+---------------+---------------+
| | Text | CSV |
+--------------------------+-------+-------+-------+-------+
| WIDE | None | 1/3 | None | 1/3 |
+--------------------------+-------+-------+-------+-------+
| Old Master | 10500 | 11100 | 12600 | 14580 |
+--------------------------+-------+-------+-------+-------+
| Old Master + 0001 + 0002 | 8900 | 10660 | 8860 | 13990 |
+--------------------------+-------+-------+-------+-------+
| | | | | |
+--------------------------+-------+-------+-------+-------+
| | Text | CSV |
+--------------------------+-------+-------+-------+-------+
| NARROW | None | 1/3 | None | 1/3 |
+--------------------------+-------+-------+-------+-------+
| Old Master | 9600 | 9700 | 9800 | 10150 |
+--------------------------+-------+-------+-------+-------+
| Old Master + 0001 + 0002 | 9300 | 9470 | 9600 | 9880 |
+--------------------------+-------+-------+-------+-------+
--
Regards,
Nazir Bilal Yavuz
Microsoft
Attachments:
[text/x-patch] v11-0001-Speed-up-COPY-FROM-text-CSV-parsing-using-SIMD.patch (9.4K, ../../CAN55FZ2DNaKCK3Kf_kHizb2pAbQvULeDYtzaiz97B_xz7YbrkQ@mail.gmail.com/2-v11-0001-Speed-up-COPY-FROM-text-CSV-parsing-using-SIMD.patch)
download | inline diff:
From 7acaeb3201ae4ae279bf8b25641bea7f8cb92cbe Mon Sep 17 00:00:00 2001
From: Nazir Bilal Yavuz <byavuz81@gmail.com>
Date: Wed, 4 Mar 2026 17:28:54 +0300
Subject: [PATCH v11] Speed up COPY FROM text/CSV parsing using SIMD
This patch disables SIMD when SIMD encounters a special character which
is neither EOF nor EOL.
Author: Shinya Kato <shinya11.kato@gmail.com>
Author: Nazir Bilal Yavuz <byavuz81@gmail.com>
Reviewed-by: Kazar Ayoub <ma_kazar@esi.dz>
Reviewed-by: Nathan Bossart <nathandbossart@gmail.com>
Reviewed-by: Neil Conway <neil.conway@gmail.com>
Reviewed-by: Andrew Dunstan <andrew@dunslane.net>
Reviewed-by: Manni Wood <manni.wood@enterprisedb.com>
Reviewed-by: Mark Wong <markwkm@gmail.com>
Discussion: https://postgr.es/m/CAOzEurSW8cNr6TPKsjrstnPfhf4QyQqB4tnPXGGe8N4e_v7Jig%40mail.gmail.com
---
src/backend/commands/copyfrom.c | 4 +
src/backend/commands/copyfromparse.c | 222 ++++++++++++++++++++++-
src/include/commands/copyfrom_internal.h | 4 +
3 files changed, 223 insertions(+), 7 deletions(-)
diff --git a/src/backend/commands/copyfrom.c b/src/backend/commands/copyfrom.c
index 2f42f55e229..2aa52810ff1 100644
--- a/src/backend/commands/copyfrom.c
+++ b/src/backend/commands/copyfrom.c
@@ -1747,6 +1747,10 @@ BeginCopyFrom(ParseState *pstate,
cstate->cur_attval = NULL;
cstate->relname_only = false;
+ /* Initialize SIMD */
+ cstate->simd_enabled = true;
+ cstate->simd_failed_first_vector = false;
+
/*
* Allocate buffers for the input pipeline.
*
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index fbd13353efc..70e1a5a0410 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -72,6 +72,7 @@
#include "miscadmin.h"
#include "pgstat.h"
#include "port/pg_bswap.h"
+#include "port/simd.h"
#include "utils/builtins.h"
#include "utils/rel.h"
@@ -158,6 +159,12 @@ static pg_attribute_always_inline bool NextCopyFromRawFieldsInternal(CopyFromSta
int *nfields,
bool is_csv);
+/* SIMD functions */
+#ifndef USE_NO_SIMD
+static bool CopyReadLineTextSIMDHelper(CopyFromState cstate, bool is_csv,
+ bool *temp_hit_eof, int *temp_input_buf_ptr);
+#endif
+
/* Low-level communications functions */
static int CopyGetData(CopyFromState cstate, void *databuf,
@@ -1310,6 +1317,182 @@ CopyReadLine(CopyFromState cstate, bool is_csv)
return result;
}
+#ifndef USE_NO_SIMD
+/*
+ * Use SIMD instructions to efficiently scan the input buffer for special
+ * characters (e.g., newline, carriage return, quote, and escape). This is
+ * faster than byte-by-byte iteration, especially on large buffers.
+ *
+ * Note that, SIMD may become slower when the input contains many special
+ * characters. To avoid this regression, we disable SIMD for the rest of the
+ * input once we encounter a special character which is neither EOF nor EOL.
+ * Also, SIMD is disabled when it encounters two consecutive short lines that
+ * SIMD can't create a full sized Vector, too.
+ */
+static bool
+CopyReadLineTextSIMDHelper(CopyFromState cstate, bool is_csv, bool *temp_hit_eof, int *temp_input_buf_ptr)
+{
+ char quotec = '\0';
+ char escapec = '\0';
+ char *copy_input_buf;
+ int input_buf_ptr;
+ int copy_buf_len;
+ bool result = false;
+ bool unique_escapec = false;
+ bool first_vector = true;
+ Vector8 nl = vector8_broadcast('\n');
+ Vector8 cr = vector8_broadcast('\r');
+ Vector8 bs = vector8_broadcast('\\');
+ Vector8 quote = vector8_broadcast(0);
+ Vector8 escape = vector8_broadcast(0);
+
+ if (is_csv)
+ {
+ quotec = cstate->opts.quote[0];
+ escapec = cstate->opts.escape[0];
+
+ quote = vector8_broadcast(quotec);
+ if (quotec != escapec)
+ {
+ unique_escapec = true;
+ escape = vector8_broadcast(escapec);
+ }
+ }
+
+ /* For a little extra speed we copy these into local variables */
+ copy_input_buf = cstate->input_buf;
+ input_buf_ptr = cstate->input_buf_index;
+ copy_buf_len = cstate->input_buf_len;
+
+ while (true)
+ {
+ /* Load more data if needed */
+ if (sizeof(Vector8) >= copy_buf_len - input_buf_ptr)
+ {
+ REFILL_LINEBUF;
+
+ CopyLoadInputBuf(cstate);
+ /* update our local variables */
+ *temp_hit_eof = cstate->input_reached_eof;
+ input_buf_ptr = cstate->input_buf_index;
+ copy_buf_len = cstate->input_buf_len;
+
+ /*
+ * If we are completely out of data, break out of the loop,
+ * reporting EOF.
+ */
+ if (INPUT_BUF_BYTES(cstate) <= 0)
+ {
+ result = true;
+ break;
+ }
+ }
+
+ if (copy_buf_len - input_buf_ptr > sizeof(Vector8))
+ {
+ Vector8 chunk;
+ Vector8 match = vector8_broadcast(0);
+
+ /* Load a chunk of data into a vector register */
+ vector8_load(&chunk, (const uint8 *) ©_input_buf[input_buf_ptr]);
+
+ if (is_csv)
+ {
+ match = vector8_or(vector8_eq(chunk, nl), vector8_eq(chunk, cr));
+ match = vector8_or(match, vector8_eq(chunk, quote));
+ if (unique_escapec)
+ match = vector8_or(match, vector8_eq(chunk, escape));
+ }
+ else
+ {
+ match = vector8_or(vector8_eq(chunk, nl), vector8_eq(chunk, cr));
+ match = vector8_or(match, vector8_eq(chunk, bs));
+ }
+
+ /* Check if we found any special characters */
+ if (vector8_is_highbit_set(match))
+ {
+ /*
+ * Found a special character. Advance up to that point and let
+ * the scalar code handle it.
+ */
+ uint32 mask;
+ int advance;
+ char c1,
+ c2;
+ bool simd_hit_eol,
+ simd_hit_eof;
+
+ mask = vector8_highbit_mask(match);
+ advance = pg_rightmost_one_pos32(mask);
+
+ input_buf_ptr += advance;
+ c1 = copy_input_buf[input_buf_ptr];
+
+ /*
+ * Since we stopped within the chunk and ((copy_buf_len -
+ * input_buf_ptr) > sizeof(Vector8)) is true,
+ * copy_input_buf[input_buf_ptr + 1] is guaranteed to be
+ * readable.
+ */
+ c2 = copy_input_buf[input_buf_ptr + 1];
+
+ simd_hit_eof = (c1 == '\\' && c2 == '.' && !is_csv);
+ simd_hit_eol = (c1 == '\r' || c1 == '\n');
+
+ /*
+ * Do not disable SIMD when we hit EOL or EOF characters. In
+ * practice, it does not matter for EOF because parsing ends
+ * there, but we keep the behavior consistent.
+ */
+ if (!(simd_hit_eof || simd_hit_eol))
+ cstate->simd_enabled = false;
+
+ /*
+ * We encountered a EOL or EOF on the first vector. This means
+ * lines are not long enough to skip fully sized vector. If
+ * this happens two times consecutively, then disable the
+ * SIMD.
+ */
+ if (first_vector)
+ {
+ if (cstate->simd_failed_first_vector)
+ cstate->simd_enabled = false;
+
+ cstate->simd_failed_first_vector = true;
+ }
+
+ break;
+ }
+ else
+ {
+ /* No special characters found, so skip the entire chunk */
+ input_buf_ptr += sizeof(Vector8);
+ first_vector = false;
+ }
+ }
+
+ /*
+ * Although we refill linebuf, there is not enough character to fill
+ * full sized vector. This doesn't mean that we encountered a line
+ * that is not enough to fill a full sized vector.
+ *
+ * Scalar code will handle the rest for this line. Then, SIMD will
+ * continue from the next line.
+ */
+ else
+ {
+ first_vector = false;
+ break;
+ }
+ }
+
+ cstate->simd_failed_first_vector = first_vector;
+ *temp_input_buf_ptr = input_buf_ptr;
+ return result;
+}
+#endif
+
/*
* CopyReadLineText - inner loop of CopyReadLine for text mode
*/
@@ -1338,6 +1521,38 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
escapec = '\0';
}
+ /* input_buf_ptr will be used in the SIMD Helper function */
+ input_buf_ptr = cstate->input_buf_index;
+
+#ifndef USE_NO_SIMD
+ /* First try to run SIMD, then continue with the scalar path */
+ if (cstate->simd_enabled)
+ {
+ int temp_input_buf_ptr = input_buf_ptr;
+ bool temp_hit_eof = false;
+
+ result = CopyReadLineTextSIMDHelper(cstate, is_csv, &temp_hit_eof,
+ &temp_input_buf_ptr);
+ input_buf_ptr = temp_input_buf_ptr;
+ hit_eof = temp_hit_eof;
+
+ /* Short exit from SIMD */
+ if (result)
+ {
+ /*
+ * Transfer any still-uncopied data to line_buf.
+ */
+ REFILL_LINEBUF;
+
+ return result;
+ }
+ }
+#endif
+
+ /* For a little extra speed we copy these into local variables */
+ copy_input_buf = cstate->input_buf;
+ copy_buf_len = cstate->input_buf_len;
+
/*
* The objective of this loop is to transfer the entire next input line
* into line_buf. Hence, we only care for detecting newlines (\r and/or
@@ -1359,14 +1574,7 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
* character to examine; any characters from input_buf_index to
* input_buf_ptr have been determined to be part of the line, but not yet
* transferred to line_buf.
- *
- * For a little extra speed within the loop, we copy input_buf and
- * input_buf_len into local variables.
*/
- copy_input_buf = cstate->input_buf;
- input_buf_ptr = cstate->input_buf_index;
- copy_buf_len = cstate->input_buf_len;
-
for (;;)
{
int prev_raw_ptr;
diff --git a/src/include/commands/copyfrom_internal.h b/src/include/commands/copyfrom_internal.h
index f892c343157..4a748df8ac8 100644
--- a/src/include/commands/copyfrom_internal.h
+++ b/src/include/commands/copyfrom_internal.h
@@ -89,6 +89,10 @@ typedef struct CopyFromStateData
const char *cur_attval; /* current att value for error messages */
bool relname_only; /* don't output line number, att, etc. */
+ /* SIMD variables */
+ bool simd_enabled;
+ bool simd_failed_first_vector;
+
/*
* Working state
*/
--
2.47.3
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 22:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-10 11:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-21 20:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Neil Conway <neil.conway@gmail.com>
2026-01-31 16:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 13:51 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 18:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 21:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-06 22:19 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 22:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-11 13:27 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-11 22:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-13 11:45 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-13 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 13:38 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-18 21:26 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 21:54 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 04:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-19 09:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 22:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-20 00:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-20 09:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-20 18:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-23 09:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 04:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-24 13:57 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 17:48 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-25 14:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-02 19:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-04 15:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
@ 2026-03-05 21:25 ` Andrew Dunstan <andrew@dunslane.net>
2026-03-06 16:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
1 sibling, 1 reply; 178+ messages in thread
From: Andrew Dunstan @ 2026-03-05 21:25 UTC (permalink / raw)
To: Nazir Bilal Yavuz <byavuz81@gmail.com>; Nathan Bossart <nathandbossart@gmail.com>; +Cc: Manni Wood <manni.wood@enterprisedb.com>; KAZAR Ayoub <ma_kazar@esi.dz>; Neil Conway <neil.conway@gmail.com>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
On 2026-03-04 We 10:15 AM, Nazir Bilal Yavuz wrote:
> Hi,
>
> On Mon, 2 Mar 2026 at 22:55, Nathan Bossart <nathandbossart@gmail.com> wrote:
>> On Wed, Feb 25, 2026 at 05:24:27PM +0300, Nazir Bilal Yavuz wrote:
>>> If anyone has any suggestions/ideas, please let me know!
> I am able to fix the problem. My first assumption was that the
> branching of SIMD code caused that problem, so I moved SIMD code to
> the CopyReadLineTextSIMDHelper() function. Then I moved this
> CopyReadLineTextSIMDHelper() to top of CopyReadLineText(), by doing
> that we won't have any branching in the non-SIMD (scalar) code path.
> This didn't solve the problem and then I realized that even though I
> disable SIMD code path with 'if (false)', there is still regression
> but if I comment all of the 'if (cstate->simd_enabled)' branch, then
> there is no regression at all.
>
> To find out more, I compared assembly outputs of both and found out
> the possible reason. What I understood is that the compiler can't
> promote a variable to register, instead these variables live in the
> stack; which is slower. Please see the two different assembly outputs:
>
> Slow code:
>
> c = copy_input_buf[input_buf_ptr++];
> db0: 48 8b 55 b8 mov -0x48(%rbp),%rdx
> db4: 48 63 c6 movslq %esi,%rax
> db7: 44 8d 66 01 lea 0x1(%rsi),%r12d
> dbb: 44 89 65 cc mov %r12d,-0x34(%rbp)
> dbf: 0f be 14 02 movsbl (%rdx,%rax,1),%edx
>
> Fast code:
>
> c = copy_input_buf[input_buf_ptr++];
> d80: 49 63 c4 movslq %r12d,%rax
> d83: 45 8d 5c 24 01 lea 0x1(%r12),%r11d
> d88: 41 0f be 04 06 movsbl (%r14,%rax,1),%eax
>
> And the reason for that is sending the address of input_buf_ptr to a
> CopyReadLineTextSIMDHelper(..., &input_buf_ptr). If I change it to
> this:
>
> int temp_input_buf_ptr = input_buf_ptr;
> CopyReadLineTextSIMDHelper(..., &temp_input_buf_ptr);
>
> Then there is no regression. However, I am still not completely sure
> if that is the same problem in the v10, I am planning to spend more
> time debugging this.
>
>> A couple of random ideas:
>>
>> * Additional inlining for callers. I looked around a little bit and didn't
>> see any great candidates, so I don't have much faith in this, but maybe
>> you'll see something I don't.
> I agree with you. CopyReadLineText() is already quite a big function.
>
>> * Disable SIMD if we are consistently getting small rows. That won't help
>> your "wide & CSV 1/3" case in all likelihood, but perhaps it'll help with
>> the regression for narrow rows described elsewhere.
> I implemented this, two consecutive small rows disables SIMD.
>
>> * Surround the variable initializations with "if (simd_enabled)".
>> Presumably compilers are smart enough to remove those in the non-SIMD paths
>> already, but it could be worth a try.
> Done.
>
>> * Add simd_enabled function parameter to CopyReadLine(),
>> NextCopyFromRawFieldsInternal(), and CopyFromTextLikeOneRow(), and do the
>> bool literal trick in CopyFrom{Text,CSV}OneRow(). That could encourage the
>> compiler to do some additional optimizations to reduce branching.
> I think we don't need this. At least the implementation with
> CopyReadLineTextSIMDHelper() doesn't need this since branching will be
> at the top and it will be once per line.
>
> I think v11 looks better compared to v10. I liked the
> CopyReadLineTextSIMDHelper() helper function. I also liked it being at
> the top of CopyReadLineText(), not being in the scalar path. This
> gives us more optimization options without affecting the scalar path.
>
> Here are the new benchmark results, I benchmarked the changes with
> both -O2 and -O3 and also both with and without 'changing
> default_toast_compression to lz4' commit (65def42b1d5). Benchmark
> results show that there is no regression and the performance
> improvement is much bigger with 65def42b1d5, it is close to 2x for
> text format and more than 2x for the csv format.
I spent some time exploring different ideas for improving this, but
found none that didn't cause regression in some cases, so good to go
from my POV.
cheers
andrew
--
Andrew Dunstan
EDB: https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 22:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-10 11:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-21 20:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Neil Conway <neil.conway@gmail.com>
2026-01-31 16:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 13:51 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 18:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 21:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-06 22:19 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 22:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-11 13:27 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-11 22:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-13 11:45 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-13 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 13:38 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-18 21:26 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 21:54 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 04:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-19 09:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 22:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-20 00:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-20 09:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-20 18:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-23 09:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 04:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-24 13:57 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 17:48 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-25 14:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-02 19:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-04 15:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-05 21:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
@ 2026-03-06 16:59 ` Manni Wood <manni.wood@enterprisedb.com>
2026-03-06 17:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
0 siblings, 1 reply; 178+ messages in thread
From: Manni Wood @ 2026-03-06 16:59 UTC (permalink / raw)
To: Andrew Dunstan <andrew@dunslane.net>; +Cc: Nazir Bilal Yavuz <byavuz81@gmail.com>; Nathan Bossart <nathandbossart@gmail.com>; KAZAR Ayoub <ma_kazar@esi.dz>; Neil Conway <neil.conway@gmail.com>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
Hello.
I ran Nazir's v11 patch on my x86 tower PC and my arm raspberry pi using
the same build I've been using: meson with "debugoptimized", which
translates to "-g -O2" gcc flags.
x86 NARROW old master (18bcdb75)
TXT : 25909.060500 ms
CSV : 28137.591250 ms
TXT with 1/3 escapes: 27794.177000 ms
CSV with 1/3 quotes: 34541.704750 ms
x86 NARROW v10
TXT : 26416.331500 ms -1.957890% regression
CSV : 25318.727500 ms 10.018142% improvement
TXT with 1/3 escapes: 28608.007500 ms -2.928061% regression
CSV with 1/3 quotes: 32805.627750 ms 5.026032% improvement
x86 NARROW v11
TXT : 27212.945750 ms -5.032545% regression
CSV : 26985.971250 ms 4.092817% improvement
TXT with 1/3 escapes: 27216.510000 ms 2.078374% improvement
CSV with 1/3 quotes: 32817.267500 ms 4.992334% improvement
x86 WIDE old master (18bcdb75)
TXT : 28778.426500 ms
CSV : 35671.908000 ms
TXT with 1/3 escapes: 32441.549750 ms
CSV with 1/3 quotes: 47024.416000 ms
x86 WIDE v10
TXT : 23067.046750 ms 19.846046% improvement
CSV : 23259.092250 ms 34.797174% improvement
TXT with 1/3 escapes: 31796.098250 ms 1.989583% improvement
CSV with 1/3 quotes: 42925.792250 ms 8.715948% improvement
x86 WIDE v11
TXT : 22571.305750 ms 21.568659% improvement
CSV : 22711.524750 ms 36.332184% improvement
TXT with 1/3 escapes: 29236.453000 ms 9.879604% improvement
CSV with 1/3 quotes: 40022.110750 ms 14.890786% improvement
arm NARROW old master (18bcdb75)
TXT : 10997.568250 ms
CSV : 10797.549000 ms
TXT with 1/3 escapes: 10299.047000 ms
CSV with 1/3 quotes: 12559.385750 ms
arm NARROW v10
TXT : 10467.816750 ms 4.816988% improvement
CSV : 9986.288000 ms 7.513381% improvement
TXT with 1/3 escapes: 10323.173750 ms -0.234262% regression
CSV with 1/3 quotes: 11843.611750 ms 5.699116% improvement
arm NARROW v11
TXT : 10340.966250 ms 5.970429% improvement
CSV : 10224.399500 ms 5.308144% improvement
TXT with 1/3 escapes: 10438.216750 ms -1.351288% regression
CSV with 1/3 quotes: 11865.934000 ms 5.521383% improvement
arm WIDE old master (18bcdb75)
TXT : 11825.771250 ms
CSV : 13907.074000 ms
TXT with 1/3 escapes: 13430.691250 ms
CSV with 1/3 quotes: 17557.954500 ms
arm WIDE v10
TXT : 9064.959000 ms 23.345727% improvement
CSV : 9019.553250 ms 35.144134% improvement
TXT with 1/3 escapes: 12344.497250 ms 8.087402% improvement
CSV with 1/3 quotes: 15495.863750 ms 11.744482% improvement
arm WIDE v11
TXT : 9001.442250 ms 23.882831% improvement
CSV : 8940.928750 ms 35.709490% improvement
TXT with 1/3 escapes: 12049.668500 ms 10.282589% improvement
CSV with 1/3 quotes: 15277.843250 ms 12.986201% improvement
Best,
-Manni
On Thu, Mar 5, 2026 at 3:25 PM Andrew Dunstan <andrew@dunslane.net> wrote:
>
> On 2026-03-04 We 10:15 AM, Nazir Bilal Yavuz wrote:
> > Hi,
> >
> > On Mon, 2 Mar 2026 at 22:55, Nathan Bossart <nathandbossart@gmail.com>
> wrote:
> >> On Wed, Feb 25, 2026 at 05:24:27PM +0300, Nazir Bilal Yavuz wrote:
> >>> If anyone has any suggestions/ideas, please let me know!
> > I am able to fix the problem. My first assumption was that the
> > branching of SIMD code caused that problem, so I moved SIMD code to
> > the CopyReadLineTextSIMDHelper() function. Then I moved this
> > CopyReadLineTextSIMDHelper() to top of CopyReadLineText(), by doing
> > that we won't have any branching in the non-SIMD (scalar) code path.
> > This didn't solve the problem and then I realized that even though I
> > disable SIMD code path with 'if (false)', there is still regression
> > but if I comment all of the 'if (cstate->simd_enabled)' branch, then
> > there is no regression at all.
> >
> > To find out more, I compared assembly outputs of both and found out
> > the possible reason. What I understood is that the compiler can't
> > promote a variable to register, instead these variables live in the
> > stack; which is slower. Please see the two different assembly outputs:
> >
> > Slow code:
> >
> > c = copy_input_buf[input_buf_ptr++];
> > db0: 48 8b 55 b8 mov -0x48(%rbp),%rdx
> > db4: 48 63 c6 movslq %esi,%rax
> > db7: 44 8d 66 01 lea 0x1(%rsi),%r12d
> > dbb: 44 89 65 cc mov %r12d,-0x34(%rbp)
> > dbf: 0f be 14 02 movsbl (%rdx,%rax,1),%edx
> >
> > Fast code:
> >
> > c = copy_input_buf[input_buf_ptr++];
> > d80: 49 63 c4 movslq %r12d,%rax
> > d83: 45 8d 5c 24 01 lea 0x1(%r12),%r11d
> > d88: 41 0f be 04 06 movsbl (%r14,%rax,1),%eax
> >
> > And the reason for that is sending the address of input_buf_ptr to a
> > CopyReadLineTextSIMDHelper(..., &input_buf_ptr). If I change it to
> > this:
> >
> > int temp_input_buf_ptr = input_buf_ptr;
> > CopyReadLineTextSIMDHelper(..., &temp_input_buf_ptr);
> >
> > Then there is no regression. However, I am still not completely sure
> > if that is the same problem in the v10, I am planning to spend more
> > time debugging this.
> >
> >> A couple of random ideas:
> >>
> >> * Additional inlining for callers. I looked around a little bit and
> didn't
> >> see any great candidates, so I don't have much faith in this, but maybe
> >> you'll see something I don't.
> > I agree with you. CopyReadLineText() is already quite a big function.
> >
> >> * Disable SIMD if we are consistently getting small rows. That won't
> help
> >> your "wide & CSV 1/3" case in all likelihood, but perhaps it'll help
> with
> >> the regression for narrow rows described elsewhere.
> > I implemented this, two consecutive small rows disables SIMD.
> >
> >> * Surround the variable initializations with "if (simd_enabled)".
> >> Presumably compilers are smart enough to remove those in the non-SIMD
> paths
> >> already, but it could be worth a try.
> > Done.
> >
> >> * Add simd_enabled function parameter to CopyReadLine(),
> >> NextCopyFromRawFieldsInternal(), and CopyFromTextLikeOneRow(), and do
> the
> >> bool literal trick in CopyFrom{Text,CSV}OneRow(). That could encourage
> the
> >> compiler to do some additional optimizations to reduce branching.
> > I think we don't need this. At least the implementation with
> > CopyReadLineTextSIMDHelper() doesn't need this since branching will be
> > at the top and it will be once per line.
> >
> > I think v11 looks better compared to v10. I liked the
> > CopyReadLineTextSIMDHelper() helper function. I also liked it being at
> > the top of CopyReadLineText(), not being in the scalar path. This
> > gives us more optimization options without affecting the scalar path.
> >
> > Here are the new benchmark results, I benchmarked the changes with
> > both -O2 and -O3 and also both with and without 'changing
> > default_toast_compression to lz4' commit (65def42b1d5). Benchmark
> > results show that there is no regression and the performance
> > improvement is much bigger with 65def42b1d5, it is close to 2x for
> > text format and more than 2x for the csv format.
>
>
> I spent some time exploring different ideas for improving this, but
> found none that didn't cause regression in some cases, so good to go
> from my POV.
>
>
> cheers
>
>
> andrew
>
>
>
> --
> Andrew Dunstan
> EDB: https://www.enterprisedb.com
>
>
--
-- Manni Wood EDB: https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 22:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-10 11:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-21 20:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Neil Conway <neil.conway@gmail.com>
2026-01-31 16:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 13:51 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 18:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 21:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-06 22:19 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 22:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-11 13:27 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-11 22:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-13 11:45 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-13 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 13:38 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-18 21:26 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 21:54 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 04:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-19 09:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 22:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-20 00:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-20 09:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-20 18:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-23 09:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 04:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-24 13:57 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 17:48 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-25 14:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-02 19:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-04 15:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-05 21:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2026-03-06 16:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
@ 2026-03-06 17:39 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-06 18:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
0 siblings, 1 reply; 178+ messages in thread
From: Nazir Bilal Yavuz @ 2026-03-06 17:39 UTC (permalink / raw)
To: Manni Wood <manni.wood@enterprisedb.com>; +Cc: Andrew Dunstan <andrew@dunslane.net>; Nathan Bossart <nathandbossart@gmail.com>; KAZAR Ayoub <ma_kazar@esi.dz>; Neil Conway <neil.conway@gmail.com>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
Hi,
On Fri, 6 Mar 2026 at 20:00, Manni Wood <manni.wood@enterprisedb.com> wrote:
>
> Hello.
>
> I ran Nazir's v11 patch on my x86 tower PC and my arm raspberry pi using the same build I've been using: meson with "debugoptimized", which translates to "-g -O2" gcc flags.
Thanks for the benchmark! The results look nice.
One question: does your benchmark include the 34dfca2934 LZ4 commit,
and is LZ4 enabled on your system?
--
Regards,
Nazir Bilal Yavuz
Microsoft
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 22:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-10 11:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-21 20:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Neil Conway <neil.conway@gmail.com>
2026-01-31 16:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 13:51 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 18:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 21:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-06 22:19 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 22:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-11 13:27 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-11 22:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-13 11:45 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-13 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 13:38 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-18 21:26 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 21:54 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 04:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-19 09:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 22:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-20 00:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-20 09:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-20 18:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-23 09:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 04:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-24 13:57 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 17:48 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-25 14:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-02 19:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-04 15:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-05 21:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2026-03-06 16:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-03-06 17:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
@ 2026-03-06 18:13 ` Manni Wood <manni.wood@enterprisedb.com>
2026-03-06 18:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
0 siblings, 1 reply; 178+ messages in thread
From: Manni Wood @ 2026-03-06 18:13 UTC (permalink / raw)
To: Nazir Bilal Yavuz <byavuz81@gmail.com>; +Cc: Andrew Dunstan <andrew@dunslane.net>; Nathan Bossart <nathandbossart@gmail.com>; KAZAR Ayoub <ma_kazar@esi.dz>; Neil Conway <neil.conway@gmail.com>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
On Fri, Mar 6, 2026 at 11:39 AM Nazir Bilal Yavuz <byavuz81@gmail.com>
wrote:
> Hi,
>
> On Fri, 6 Mar 2026 at 20:00, Manni Wood <manni.wood@enterprisedb.com>
> wrote:
> >
> > Hello.
> >
> > I ran Nazir's v11 patch on my x86 tower PC and my arm raspberry pi using
> the same build I've been using: meson with "debugoptimized", which
> translates to "-g -O2" gcc flags.
>
> Thanks for the benchmark! The results look nice.
>
> One question: does your benchmark include the 34dfca2934 LZ4 commit,
> and is LZ4 enabled on your system?
>
> --
> Regards,
> Nazir Bilal Yavuz
> Microsoft
>
Hello, Nazir!
When I ran `meson setup build --buildtype=debugoptimized` on both my x86
machine and my arm machine, the response on both was:
"External libraries
" lz4 : NO"
However, I did not remove commit 34dfca2934 from any of my Postgres builds;
I left that commit in place.
Let me know if that helps!
And I agree, the results look nice.
Best,
-Manni
--
-- Manni Wood EDB: https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 22:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-10 11:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-21 20:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Neil Conway <neil.conway@gmail.com>
2026-01-31 16:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 13:51 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 18:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 21:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-06 22:19 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 22:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-11 13:27 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-11 22:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-13 11:45 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-13 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 13:38 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-18 21:26 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 21:54 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 04:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-19 09:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 22:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-20 00:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-20 09:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-20 18:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-23 09:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 04:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-24 13:57 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 17:48 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-25 14:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-02 19:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-04 15:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-05 21:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2026-03-06 16:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-03-06 17:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-06 18:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
@ 2026-03-06 18:55 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-06 21:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
0 siblings, 1 reply; 178+ messages in thread
From: Nazir Bilal Yavuz @ 2026-03-06 18:55 UTC (permalink / raw)
To: Manni Wood <manni.wood@enterprisedb.com>; +Cc: Andrew Dunstan <andrew@dunslane.net>; Nathan Bossart <nathandbossart@gmail.com>; KAZAR Ayoub <ma_kazar@esi.dz>; Neil Conway <neil.conway@gmail.com>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
Hi Manni!
On Fri, 6 Mar 2026 at 21:13, Manni Wood <manni.wood@enterprisedb.com> wrote:
>
> When I ran `meson setup build --buildtype=debugoptimized` on both my x86 machine and my arm machine, the response on both was:
>
> "External libraries
> " lz4 : NO"
>
> However, I did not remove commit 34dfca2934 from any of my Postgres builds; I left that commit in place.
>
> Let me know if that helps!
That definitely helps, thanks! If you have a chance, could you also
run the benchmark with LZ4 enabled? I expect you may see significantly
better performance, similar to what I observed.
--
Regards,
Nazir Bilal Yavuz
Microsoft
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 22:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-10 11:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-21 20:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Neil Conway <neil.conway@gmail.com>
2026-01-31 16:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 13:51 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 18:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 21:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-06 22:19 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 22:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-11 13:27 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-11 22:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-13 11:45 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-13 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 13:38 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-18 21:26 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 21:54 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 04:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-19 09:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 22:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-20 00:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-20 09:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-20 18:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-23 09:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 04:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-24 13:57 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 17:48 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-25 14:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-02 19:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-04 15:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-05 21:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2026-03-06 16:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-03-06 17:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-06 18:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-03-06 18:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
@ 2026-03-06 21:25 ` Manni Wood <manni.wood@enterprisedb.com>
2026-03-06 23:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
0 siblings, 1 reply; 178+ messages in thread
From: Manni Wood @ 2026-03-06 21:25 UTC (permalink / raw)
To: Nazir Bilal Yavuz <byavuz81@gmail.com>; +Cc: Andrew Dunstan <andrew@dunslane.net>; Nathan Bossart <nathandbossart@gmail.com>; KAZAR Ayoub <ma_kazar@esi.dz>; Neil Conway <neil.conway@gmail.com>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
On Fri, Mar 6, 2026 at 12:55 PM Nazir Bilal Yavuz <byavuz81@gmail.com>
wrote:
> Hi Manni!
>
> On Fri, 6 Mar 2026 at 21:13, Manni Wood <manni.wood@enterprisedb.com>
> wrote:
> >
> > When I ran `meson setup build --buildtype=debugoptimized` on both my x86
> machine and my arm machine, the response on both was:
> >
> > "External libraries
> > " lz4 : NO"
> >
> > However, I did not remove commit 34dfca2934 from any of my Postgres
> builds; I left that commit in place.
> >
> > Let me know if that helps!
>
> That definitely helps, thanks! If you have a chance, could you also
> run the benchmark with LZ4 enabled? I expect you may see significantly
> better performance, similar to what I observed.
>
> --
> Regards,
> Nazir Bilal Yavuz
> Microsoft
>
Hi, Nazir.
Well, golly! Look at these numbers. Old master with no lz4, your v11 patch
with no lz4, and then your v11 patch with lz4 compiled in.
(Aside: I assume everything is good for my lz4 build after installing the
lz4 dev library and seeing this with my meson config:
" External libraries
" lz4 : YES 1.9.4"
and this from the db itself:
$ ./bin/psql -U mwood -d postgres -c 'show default_toast_compression'
default_toast_compression
---------------------------
lz4
)
x86 NARROW old master
TXT : 25909.060500 ms
CSV : 28137.591250 ms
TXT with 1/3 escapes: 27794.177000 ms
CSV with 1/3 quotes: 34541.704750 ms
x86 NARROW v11
TXT : 27212.945750 ms -5.032545% regression
CSV : 26985.971250 ms 4.092817% improvement
TXT with 1/3 escapes: 27216.510000 ms 2.078374% improvement
CSV with 1/3 quotes: 32817.267500 ms 4.992334% improvement
x86 NARROW v11 lz4
TXT : 26471.776500 ms -2.171889% regression
CSV : 25607.026250 ms 8.993538% improvement
TXT with 1/3 escapes: 28628.729750 ms -3.002617% regression
CSV with 1/3 quotes: 34729.006750 ms -0.542249% regression
x86 WIDE old master
TXT : 28778.426500 ms
CSV : 35671.908000 ms
TXT with 1/3 escapes: 32441.549750 ms
CSV with 1/3 quotes: 47024.416000 ms
x86 WIDE v11
TXT : 22571.305750 ms 21.568659% improvement
CSV : 22711.524750 ms 36.332184% improvement
TXT with 1/3 escapes: 29236.453000 ms 9.879604% improvement
CSV with 1/3 quotes: 40022.110750 ms 14.890786% improvement
x86 WIDE v11 lz4
TXT : 8032.912750 ms 72.087033% improvement
CSV : 8047.098000 ms 77.441358% improvement
TXT with 1/3 escapes: 15428.139500 ms 52.443272% improvement
CSV with 1/3 quotes: 27517.084500 ms 41.483410% improvement
arm NARROW old master
TXT : 10997.568250 ms
CSV : 10797.549000 ms
TXT with 1/3 escapes: 10299.047000 ms
CSV with 1/3 quotes: 12559.385750 ms
arm NARROW v11
TXT : 10340.966250 ms 5.970429% improvement
CSV : 10224.399500 ms 5.308144% improvement
TXT with 1/3 escapes: 10438.216750 ms -1.351288% regression
CSV with 1/3 quotes: 11865.934000 ms 5.521383% improvement
arm NARROW v11 lz4
TXT : 9783.737000 ms 11.037270% improvement
CSV : 10122.890750 ms 6.248254% improvement
TXT with 1/3 escapes: 10298.780250 ms 0.002590% improvement
CSV with 1/3 quotes: 11738.992250 ms 6.532115% improvement
arm WIDE old master
TXT : 11825.771250 ms
CSV : 13907.074000 ms
TXT with 1/3 escapes: 13430.691250 ms
CSV with 1/3 quotes: 17557.954500 ms
arm WIDE v11
TXT : 9001.442250 ms 23.882831% improvement
CSV : 8940.928750 ms 35.709490% improvement
TXT with 1/3 escapes: 12049.668500 ms 10.282589% improvement
CSV with 1/3 quotes: 15277.843250 ms 12.986201% improvement
arm WIDE v11 lz4
TXT : 3186.825500 ms 73.051859% improvement
CSV : 3142.526500 ms 77.403396% improvement
TXT with 1/3 escapes: 6180.176000 ms 53.984677% improvement
CSV with 1/3 quotes: 9460.505500 ms 46.118407% improvement
Cheers,
-Manni
--
-- Manni Wood EDB: https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 22:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-10 11:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-21 20:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Neil Conway <neil.conway@gmail.com>
2026-01-31 16:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 13:51 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 18:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 21:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-06 22:19 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 22:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-11 13:27 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-11 22:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-13 11:45 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-13 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 13:38 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-18 21:26 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 21:54 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 04:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-19 09:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 22:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-20 00:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-20 09:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-20 18:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-23 09:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 04:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-24 13:57 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 17:48 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-25 14:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-02 19:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-04 15:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-05 21:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2026-03-06 16:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-03-06 17:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-06 18:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-03-06 18:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-06 21:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
@ 2026-03-06 23:13 ` Nathan Bossart <nathandbossart@gmail.com>
2026-03-06 23:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
0 siblings, 1 reply; 178+ messages in thread
From: Nathan Bossart @ 2026-03-06 23:13 UTC (permalink / raw)
To: Manni Wood <manni.wood@enterprisedb.com>; +Cc: Nazir Bilal Yavuz <byavuz81@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; KAZAR Ayoub <ma_kazar@esi.dz>; Neil Conway <neil.conway@gmail.com>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
On Fri, Mar 06, 2026 at 03:25:46PM -0600, Manni Wood wrote:
> Well, golly! Look at these numbers. Old master with no lz4, your v11 patch
> with no lz4, and then your v11 patch with lz4 compiled in.
I'm appreciative of all the benchmarking that you and others are doing, but
wouldn't we be more interested in the difference between "old master with
lz4" and "v11 with lz4"? Else, we have multiple variables in play.
--
nathan
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 22:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-10 11:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-21 20:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Neil Conway <neil.conway@gmail.com>
2026-01-31 16:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 13:51 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 18:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 21:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-06 22:19 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 22:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-11 13:27 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-11 22:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-13 11:45 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-13 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 13:38 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-18 21:26 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 21:54 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 04:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-19 09:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 22:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-20 00:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-20 09:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-20 18:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-23 09:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 04:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-24 13:57 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 17:48 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-25 14:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-02 19:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-04 15:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-05 21:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2026-03-06 16:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-03-06 17:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-06 18:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-03-06 18:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-06 21:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-03-06 23:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
@ 2026-03-06 23:31 ` KAZAR Ayoub <ma_kazar@esi.dz>
2026-03-08 10:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
0 siblings, 1 reply; 178+ messages in thread
From: KAZAR Ayoub @ 2026-03-06 23:31 UTC (permalink / raw)
To: Nathan Bossart <nathandbossart@gmail.com>; +Cc: Manni Wood <manni.wood@enterprisedb.com>; Nazir Bilal Yavuz <byavuz81@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Neil Conway <neil.conway@gmail.com>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
Hi,
On Sat, Mar 7, 2026 at 12:13 AM Nathan Bossart <nathandbossart@gmail.com>
wrote:
> On Fri, Mar 06, 2026 at 03:25:46PM -0600, Manni Wood wrote:
> > Well, golly! Look at these numbers. Old master with no lz4, your v11
> patch
> > with no lz4, and then your v11 patch with lz4 compiled in.
>
> I'm appreciative of all the benchmarking that you and others are doing, but
> wouldn't we be more interested in the difference between "old master with
> lz4" and "v11 with lz4"? Else, we have multiple variables in play.
>
Yes I agree because the lz4 effect doesn't prove anything for the SIMD
patch itself right ? So basically a comparison for the SIMD effect should
be "master with/out lz4 vs patched with/out lz4, respectively and nothing
more!", is this correct ?
Regards,
Ayoub
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 22:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-10 11:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-21 20:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Neil Conway <neil.conway@gmail.com>
2026-01-31 16:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 13:51 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 18:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 21:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-06 22:19 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 22:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-11 13:27 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-11 22:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-13 11:45 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-13 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 13:38 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-18 21:26 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 21:54 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 04:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-19 09:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 22:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-20 00:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-20 09:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-20 18:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-23 09:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 04:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-24 13:57 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 17:48 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-25 14:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-02 19:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-04 15:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-05 21:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2026-03-06 16:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-03-06 17:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-06 18:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-03-06 18:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-06 21:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-03-06 23:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-06 23:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
@ 2026-03-08 10:31 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-08 19:45 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
0 siblings, 1 reply; 178+ messages in thread
From: Nazir Bilal Yavuz @ 2026-03-08 10:31 UTC (permalink / raw)
To: KAZAR Ayoub <ma_kazar@esi.dz>; +Cc: Nathan Bossart <nathandbossart@gmail.com>; Manni Wood <manni.wood@enterprisedb.com>; Andrew Dunstan <andrew@dunslane.net>; Neil Conway <neil.conway@gmail.com>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
Hi,
On Sat, 7 Mar 2026 at 02:31, KAZAR Ayoub <ma_kazar@esi.dz> wrote:
>
> On Sat, Mar 7, 2026 at 12:13 AM Nathan Bossart <nathandbossart@gmail.com> wrote:
>>
>> On Fri, Mar 06, 2026 at 03:25:46PM -0600, Manni Wood wrote:
>> > Well, golly! Look at these numbers. Old master with no lz4, your v11 patch
>> > with no lz4, and then your v11 patch with lz4 compiled in.
>>
>> I'm appreciative of all the benchmarking that you and others are doing, but
>> wouldn't we be more interested in the difference between "old master with
>> lz4" and "v11 with lz4"? Else, we have multiple variables in play.
>
> Yes I agree because the lz4 effect doesn't prove anything for the SIMD patch itself right ? So basically a comparison for the SIMD effect should be "master with/out lz4 vs patched with/out lz4, respectively and nothing more!", is this correct ?
Yes, I think 'master with/out lz4 vs patched with/out lz4,
respectively' is enough to determine the effect of the SIMD patch.
--
Regards,
Nazir Bilal Yavuz
Microsoft
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 22:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-10 11:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-21 20:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Neil Conway <neil.conway@gmail.com>
2026-01-31 16:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 13:51 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 18:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 21:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-06 22:19 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 22:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-11 13:27 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-11 22:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-13 11:45 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-13 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 13:38 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-18 21:26 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 21:54 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 04:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-19 09:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 22:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-20 00:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-20 09:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-20 18:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-23 09:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 04:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-24 13:57 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 17:48 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-25 14:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-02 19:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-04 15:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-05 21:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2026-03-06 16:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-03-06 17:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-06 18:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-03-06 18:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-06 21:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-03-06 23:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-06 23:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-03-08 10:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
@ 2026-03-08 19:45 ` Manni Wood <manni.wood@enterprisedb.com>
2026-03-09 08:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
0 siblings, 1 reply; 178+ messages in thread
From: Manni Wood @ 2026-03-08 19:45 UTC (permalink / raw)
To: Nazir Bilal Yavuz <byavuz81@gmail.com>; +Cc: KAZAR Ayoub <ma_kazar@esi.dz>; Nathan Bossart <nathandbossart@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Neil Conway <neil.conway@gmail.com>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
On Sun, Mar 8, 2026 at 5:31 AM Nazir Bilal Yavuz <byavuz81@gmail.com> wrote:
> Hi,
>
> On Sat, 7 Mar 2026 at 02:31, KAZAR Ayoub <ma_kazar@esi.dz> wrote:
> >
> > On Sat, Mar 7, 2026 at 12:13 AM Nathan Bossart <nathandbossart@gmail.com>
> wrote:
> >>
> >> On Fri, Mar 06, 2026 at 03:25:46PM -0600, Manni Wood wrote:
> >> > Well, golly! Look at these numbers. Old master with no lz4, your v11
> patch
> >> > with no lz4, and then your v11 patch with lz4 compiled in.
> >>
> >> I'm appreciative of all the benchmarking that you and others are doing,
> but
> >> wouldn't we be more interested in the difference between "old master
> with
> >> lz4" and "v11 with lz4"? Else, we have multiple variables in play.
> >
> > Yes I agree because the lz4 effect doesn't prove anything for the SIMD
> patch itself right ? So basically a comparison for the SIMD effect should
> be "master with/out lz4 vs patched with/out lz4, respectively and nothing
> more!", is this correct ?
>
> Yes, I think 'master with/out lz4 vs patched with/out lz4,
> respectively' is enough to determine the effect of the SIMD patch.
>
> --
> Regards,
> Nazir Bilal Yavuz
> Microsoft
>
Hello!
As requested, here are some numbers based on the latest master but with the
copy code inlining excised (`git revert
dc592a41557b072178f1798700bf9c69cd8e4235`), compared to master with copy
code inlining left in place and the v11 patch applied.
Both results have lz4 compression in place.
I have not run numbers without lz4. I assume I could use the two postgres
instances that I have compiled with lz4, but just set
`default_toast_compression = pglz` in postgesql.conf for both instances.
Let me know if that is a mistaken assumption on my part.
arm NARROW master without inline with lz4
TXT : 10362.799500 ms
CSV : 10288.791000 ms
TXT with 1/3 escapes: 10411.416250 ms
CSV with 1/3 quotes: 12318.385750 ms
arm NARROW master with inline with lz4 with v11patch
TXT : 10317.125750 ms 0.440747% improvement
CSV : 10418.020250 ms -1.256020% regression
TXT with 1/3 escapes: 10188.319500 ms 2.142809% improvement
CSV with 1/3 quotes: 12032.964500 ms 2.317035% improvement
arm WIDE master without inline with lz4
TXT : 5608.834500 ms
CSV : 8115.155000 ms
TXT with 1/3 escapes: 7037.290500 ms
CSV with 1/3 quotes: 10894.615750 ms
arm WIDE master with inline with lz4 with v11patch
TXT : 3190.268750 ms 43.120647% improvement
CSV : 3135.177000 ms 61.366394% improvement
TXT with 1/3 escapes: 6373.746750 ms 9.428966% improvement
CSV with 1/3 quotes: 10336.763500 ms 5.120440% improvement
x86 NARROW-master-without-inline-with-lz4.log
TXT : 26701.079250 ms
CSV : 26492.235500 ms
TXT with 1/3 escapes: 28590.508250 ms
CSV with 1/3 quotes: 34876.742750 ms
x86 NARROW-master-with-inline-with-lz4-with-v11patch.log
TXT : 26511.747750 ms 0.709078% improvement
CSV : 26261.269750 ms 0.871824% improvement
TXT with 1/3 escapes: 27702.964750 ms 3.104329% improvement
CSV with 1/3 quotes: 32339.393000 ms 7.275191% improvement
x86 WIDE-master-without-inline-with-lz4.log
TXT : 14485.563250 ms
CSV : 21392.582000 ms
TXT with 1/3 escapes: 18081.514750 ms
CSV with 1/3 quotes: 32547.086250 ms
x86 WIDE-master-with-inline-with-lz4-with-v11patch.log
TXT : 8080.378250 ms 44.217714% improvement
CSV : 8283.723000 ms 61.277591% improvement
TXT with 1/3 escapes: 15054.111000 ms 16.743087% improvement
CSV with 1/3 quotes: 25668.009750 ms 21.135768% improvement
--
-- Manni Wood EDB: https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 22:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-10 11:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-21 20:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Neil Conway <neil.conway@gmail.com>
2026-01-31 16:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 13:51 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 18:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 21:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-06 22:19 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 22:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-11 13:27 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-11 22:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-13 11:45 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-13 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 13:38 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-18 21:26 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 21:54 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 04:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-19 09:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 22:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-20 00:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-20 09:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-20 18:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-23 09:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 04:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-24 13:57 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 17:48 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-25 14:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-02 19:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-04 15:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-05 21:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2026-03-06 16:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-03-06 17:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-06 18:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-03-06 18:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-06 21:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-03-06 23:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-06 23:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-03-08 10:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-08 19:45 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
@ 2026-03-09 08:10 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-09 13:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
0 siblings, 1 reply; 178+ messages in thread
From: Nazir Bilal Yavuz @ 2026-03-09 08:10 UTC (permalink / raw)
To: Manni Wood <manni.wood@enterprisedb.com>; +Cc: KAZAR Ayoub <ma_kazar@esi.dz>; Nathan Bossart <nathandbossart@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Neil Conway <neil.conway@gmail.com>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
Hi,
On Sun, 8 Mar 2026 at 22:45, Manni Wood <manni.wood@enterprisedb.com> wrote:
>
> As requested, here are some numbers based on the latest master but with the copy code inlining excised (`git revert dc592a41557b072178f1798700bf9c69cd8e4235`), compared to master with copy code inlining left in place and the v11 patch applied.
> Both results have lz4 compression in place.
Thank you for the benchmark!
> I have not run numbers without lz4. I assume I could use the two postgres instances that I have compiled with lz4, but just set `default_toast_compression = pglz` in postgesql.conf for both instances. Let me know if that is a mistaken assumption on my part.
I am a bit confused. Are you asking that for the current benchmark you
shared or future benchmarks? I assume your current benchmark has
'default_toast_compression = lz4' because your benchmark results are
very similar to my benchmark with 'default_toast_compression = lz4'
but I just wanted to make sure.
What you said about editing postgresql.conf is correct but you need to
make this change before creating the Postgres instance with 'pg_ctl
... start' command, otherwise it won't have an effect and you need to
restart the instance to see the effect. Also, If you want to benchmark
without lz4 change, you can just use the "SET
default_toast_compression to 'pglz';" command in psql, then you don't
need to edit postgresql.conf. Please note that this will affect only
the psql instance you typed the command. To make things easier, you
can run the 'SHOW default_toast_compression;' command to see the
current value of 'default_toast_compression'.
--
Regards,
Nazir Bilal Yavuz
Microsoft
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 22:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-10 11:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-21 20:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Neil Conway <neil.conway@gmail.com>
2026-01-31 16:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 13:51 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 18:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 21:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-06 22:19 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 22:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-11 13:27 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-11 22:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-13 11:45 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-13 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 13:38 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-18 21:26 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 21:54 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 04:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-19 09:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 22:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-20 00:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-20 09:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-20 18:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-23 09:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 04:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-24 13:57 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 17:48 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-25 14:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-02 19:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-04 15:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-05 21:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2026-03-06 16:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-03-06 17:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-06 18:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-03-06 18:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-06 21:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-03-06 23:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-06 23:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-03-08 10:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-08 19:45 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-03-09 08:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
@ 2026-03-09 13:31 ` Manni Wood <manni.wood@enterprisedb.com>
2026-03-09 13:43 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
0 siblings, 1 reply; 178+ messages in thread
From: Manni Wood @ 2026-03-09 13:31 UTC (permalink / raw)
To: Nazir Bilal Yavuz <byavuz81@gmail.com>; +Cc: KAZAR Ayoub <ma_kazar@esi.dz>; Nathan Bossart <nathandbossart@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Neil Conway <neil.conway@gmail.com>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
On Mon, Mar 9, 2026 at 3:10 AM Nazir Bilal Yavuz <byavuz81@gmail.com> wrote:
> Hi,
>
> On Sun, 8 Mar 2026 at 22:45, Manni Wood <manni.wood@enterprisedb.com>
> wrote:
> >
> > As requested, here are some numbers based on the latest master but with
> the copy code inlining excised (`git revert
> dc592a41557b072178f1798700bf9c69cd8e4235`), compared to master with copy
> code inlining left in place and the v11 patch applied.
> > Both results have lz4 compression in place.
>
> Thank you for the benchmark!
>
> > I have not run numbers without lz4. I assume I could use the two
> postgres instances that I have compiled with lz4, but just set
> `default_toast_compression = pglz` in postgesql.conf for both instances.
> Let me know if that is a mistaken assumption on my part.
>
> I am a bit confused. Are you asking that for the current benchmark you
> shared or future benchmarks? I assume your current benchmark has
> 'default_toast_compression = lz4' because your benchmark results are
> very similar to my benchmark with 'default_toast_compression = lz4'
> but I just wanted to make sure.
>
> What you said about editing postgresql.conf is correct but you need to
> make this change before creating the Postgres instance with 'pg_ctl
> ... start' command, otherwise it won't have an effect and you need to
> restart the instance to see the effect. Also, If you want to benchmark
> without lz4 change, you can just use the "SET
> default_toast_compression to 'pglz';" command in psql, then you don't
> need to edit postgresql.conf. Please note that this will affect only
> the psql instance you typed the command. To make things easier, you
> can run the 'SHOW default_toast_compression;' command to see the
> current value of 'default_toast_compression'.
>
> --
> Regards,
> Nazir Bilal Yavuz
> Microsoft
>
Hello, Nazir!
I was being too brief.
The benchmarks I shared were absolutely with lz4 compiled in
and 'default_toast_compression = lz4' set in postgresql.conf for every
postgres instance I tested with. (Furthermore, I ran `show
default_toast_compression` via `psql` on each postgres instance to be
sure 'default_toast_compression = lz4' was really set!)
Also, all were compiled using meson using `debugoptimized` which results in
`-g -O2`.
So those are the benchmarks that I shared.
OK, so my final question, hopefully clarified: If I run additional
benchmarks where pglz is used for default_toast_compression, is it enough
to use the instances I have already compiled with lz4 in them, but
with 'default_toast_compression = pglz` explicitly set in postgresql.conf
in a brand new data dir created by initdb? (In other words, existing data
dir deleted, then initdb run to make a new data dir, then postgresql.conf
edited to ensure 'default_toast_compression = pglz` explicitly set, then
and only then starting up the cluster for the first time... and finally
verifying via `show default_toast_compression` for good measure.)
Or should I re-compile with the lz4-is-now-the-default commit completely
excised?
Thanks so much!
-Manni
--
-- Manni Wood EDB: https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 22:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-10 11:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-21 20:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Neil Conway <neil.conway@gmail.com>
2026-01-31 16:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 13:51 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 18:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 21:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-06 22:19 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 22:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-11 13:27 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-11 22:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-13 11:45 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-13 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 13:38 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-18 21:26 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 21:54 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 04:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-19 09:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 22:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-20 00:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-20 09:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-20 18:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-23 09:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 04:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-24 13:57 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 17:48 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-25 14:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-02 19:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-04 15:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-05 21:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2026-03-06 16:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-03-06 17:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-06 18:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-03-06 18:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-06 21:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-03-06 23:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-06 23:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-03-08 10:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-08 19:45 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-03-09 08:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-09 13:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
@ 2026-03-09 13:43 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
0 siblings, 0 replies; 178+ messages in thread
From: Nazir Bilal Yavuz @ 2026-03-09 13:43 UTC (permalink / raw)
To: Manni Wood <manni.wood@enterprisedb.com>; +Cc: KAZAR Ayoub <ma_kazar@esi.dz>; Nathan Bossart <nathandbossart@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Neil Conway <neil.conway@gmail.com>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
Hi Manni!
On Mon, 9 Mar 2026 at 16:31, Manni Wood <manni.wood@enterprisedb.com> wrote:
>
> I was being too brief.
>
> The benchmarks I shared were absolutely with lz4 compiled in and 'default_toast_compression = lz4' set in postgresql.conf for every postgres instance I tested with. (Furthermore, I ran `show default_toast_compression` via `psql` on each postgres instance to be sure 'default_toast_compression = lz4' was really set!)
>
> Also, all were compiled using meson using `debugoptimized` which results in `-g -O2`.
>
> So those are the benchmarks that I shared.
Thanks for the clarification.
> OK, so my final question, hopefully clarified: If I run additional benchmarks where pglz is used for default_toast_compression, is it enough to use the instances I have already compiled with lz4 in them, but with 'default_toast_compression = pglz` explicitly set in postgresql.conf in a brand new data dir created by initdb? (In other words, existing data dir deleted, then initdb run to make a new data dir, then postgresql.conf edited to ensure 'default_toast_compression = pglz` explicitly set, then and only then starting up the cluster for the first time... and finally verifying via `show default_toast_compression` for good measure.)
>
> Or should I re-compile with the lz4-is-now-the-default commit completely excised?
Yes, it is clear now; thanks. You don't need to compile without the
lz4-is-now-the-default commit. You can compile with lz4 commit and set
the 'default_toast_compression = pglz' in the postgresql.conf like you
said. This should be enough.
--
Regards,
Nazir Bilal Yavuz
Microsoft
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 22:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-10 11:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-21 20:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Neil Conway <neil.conway@gmail.com>
2026-01-31 16:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 13:51 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 18:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 21:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-06 22:19 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 22:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-11 13:27 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-11 22:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-13 11:45 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-13 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 13:38 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-18 21:26 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 21:54 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 04:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-19 09:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 22:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-20 00:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-20 09:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-20 18:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-23 09:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 04:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-24 13:57 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 17:48 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-25 14:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-02 19:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-04 15:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
@ 2026-03-09 18:25 ` Nathan Bossart <nathandbossart@gmail.com>
2026-03-10 02:30 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-03-10 12:35 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
1 sibling, 2 replies; 178+ messages in thread
From: Nathan Bossart @ 2026-03-09 18:25 UTC (permalink / raw)
To: Nazir Bilal Yavuz <byavuz81@gmail.com>; +Cc: Manni Wood <manni.wood@enterprisedb.com>; KAZAR Ayoub <ma_kazar@esi.dz>; Neil Conway <neil.conway@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
On Wed, Mar 04, 2026 at 06:15:53PM +0300, Nazir Bilal Yavuz wrote:
> +#ifndef USE_NO_SIMD
> +static bool CopyReadLineTextSIMDHelper(CopyFromState cstate, bool is_csv,
> + bool *temp_hit_eof, int *temp_input_buf_ptr);
> +#endif
Should we inline this, too?
> + /*
> + * Do not disable SIMD when we hit EOL or EOF characters. In
> + * practice, it does not matter for EOF because parsing ends
> + * there, but we keep the behavior consistent.
> + */
> + if (!(simd_hit_eof || simd_hit_eol))
> + cstate->simd_enabled = false;
nitpick: I would personally avoid disabling it for EOF. It probably
doesn't amount to much, but I don't see any point in the extra
complexity/work solely for consistency.
> + /*
> + * We encountered a EOL or EOF on the first vector. This means
> + * lines are not long enough to skip fully sized vector. If
> + * this happens two times consecutively, then disable the
> + * SIMD.
> + */
> + if (first_vector)
> + {
> + if (cstate->simd_failed_first_vector)
> + cstate->simd_enabled = false;
> +
> + cstate->simd_failed_first_vector = true;
> + }
The first time I saw this, my mind immediately went to the extreme case
where this likely regresses: alternating long and short lines. We might
just want to disable it the first time we see a short line, like we do for
special characters. This is another thing that we can improve
independently later on.
> + /* First try to run SIMD, then continue with the scalar path */
> + if (cstate->simd_enabled)
> + {
> + int temp_input_buf_ptr = input_buf_ptr;
> + bool temp_hit_eof = false;
> +
> + result = CopyReadLineTextSIMDHelper(cstate, is_csv, &temp_hit_eof,
> + &temp_input_buf_ptr);
> + input_buf_ptr = temp_input_buf_ptr;
> + hit_eof = temp_hit_eof;
Given CopyReadLineTextSIMDHelper() doesn't have too much duplicated code,
moving the SIMD stuff to its own function is nice. The temp variables seem
a bit too magical to me, though. If those really make a difference, IMHO
there ought to be a big comment explaining why.
--
nathan
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 22:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-10 11:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-21 20:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Neil Conway <neil.conway@gmail.com>
2026-01-31 16:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 13:51 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 18:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 21:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-06 22:19 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 22:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-11 13:27 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-11 22:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-13 11:45 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-13 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 13:38 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-18 21:26 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 21:54 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 04:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-19 09:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 22:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-20 00:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-20 09:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-20 18:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-23 09:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 04:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-24 13:57 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 17:48 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-25 14:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-02 19:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-04 15:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-09 18:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
@ 2026-03-10 02:30 ` Manni Wood <manni.wood@enterprisedb.com>
2026-03-10 11:42 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
1 sibling, 1 reply; 178+ messages in thread
From: Manni Wood @ 2026-03-10 02:30 UTC (permalink / raw)
To: Nathan Bossart <nathandbossart@gmail.com>; +Cc: Nazir Bilal Yavuz <byavuz81@gmail.com>; KAZAR Ayoub <ma_kazar@esi.dz>; Neil Conway <neil.conway@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
On Mon, Mar 9, 2026 at 1:25 PM Nathan Bossart <nathandbossart@gmail.com>
wrote:
> On Wed, Mar 04, 2026 at 06:15:53PM +0300, Nazir Bilal Yavuz wrote:
> > +#ifndef USE_NO_SIMD
> > +static bool CopyReadLineTextSIMDHelper(CopyFromState cstate, bool
> is_csv,
> > +
> bool *temp_hit_eof, int *temp_input_buf_ptr);
> > +#endif
>
> Should we inline this, too?
>
> > + /*
> > + * Do not disable SIMD when we hit EOL or
> EOF characters. In
> > + * practice, it does not matter for EOF
> because parsing ends
> > + * there, but we keep the behavior
> consistent.
> > + */
> > + if (!(simd_hit_eof || simd_hit_eol))
> > + cstate->simd_enabled = false;
>
> nitpick: I would personally avoid disabling it for EOF. It probably
> doesn't amount to much, but I don't see any point in the extra
> complexity/work solely for consistency.
>
> > + /*
> > + * We encountered a EOL or EOF on the
> first vector. This means
> > + * lines are not long enough to skip fully
> sized vector. If
> > + * this happens two times consecutively,
> then disable the
> > + * SIMD.
> > + */
> > + if (first_vector)
> > + {
> > + if
> (cstate->simd_failed_first_vector)
> > + cstate->simd_enabled =
> false;
> > +
> > + cstate->simd_failed_first_vector =
> true;
> > + }
>
> The first time I saw this, my mind immediately went to the extreme case
> where this likely regresses: alternating long and short lines. We might
> just want to disable it the first time we see a short line, like we do for
> special characters. This is another thing that we can improve
> independently later on.
>
> > + /* First try to run SIMD, then continue with the scalar path */
> > + if (cstate->simd_enabled)
> > + {
> > + int temp_input_buf_ptr = input_buf_ptr;
> > + bool temp_hit_eof = false;
> > +
> > + result = CopyReadLineTextSIMDHelper(cstate, is_csv,
> &temp_hit_eof,
> > +
> &temp_input_buf_ptr);
> > + input_buf_ptr = temp_input_buf_ptr;
> > + hit_eof = temp_hit_eof;
>
> Given CopyReadLineTextSIMDHelper() doesn't have too much duplicated code,
> moving the SIMD stuff to its own function is nice. The temp variables seem
> a bit too magical to me, though. If those really make a difference, IMHO
> there ought to be a big comment explaining why.
>
> --
> nathan
>
Here are some benchmarks showing what performance will look like for users
who continue to use default_toast_compression = pglz.
all compiled by meson with debugoptimized (-g -O2)
arm NARROW master without inline (git revert
dc592a41557b072178f1798700bf9c69cd8e4235) default_toast_compression = pglz
TXT : 10055.141000 ms
CSV : 10549.174500 ms
TXT with 1/3 escapes: 10213.864750 ms
CSV with 1/3 quotes: 12188.039000 ms
arm NARROW master with inline with v11patch default_toast_compression = pglz
TXT : 10070.153750 ms -0.149304% regression
CSV : 10161.348750 ms 3.676361% improvement
TXT with 1/3 escapes: 10618.005000 ms -3.956781% regression
CSV with 1/3 quotes: 12279.366250 ms -0.749319% regression
arm WIDE master without inline (git revert
dc592a41557b072178f1798700bf9c69cd8e4235) default_toast_compression = pglz
TXT : 11355.602750 ms
CSV : 13893.110500 ms
TXT with 1/3 escapes: 12872.690500 ms
CSV with 1/3 quotes: 16722.262500 ms
arm WIDE master with inline with v11patch default_toast_compression = pglz
TXT : 9001.007250 ms 20.735099% improvement
CSV : 8988.679750 ms 35.301171% improvement
TXT with 1/3 escapes: 12191.137000 ms 5.294569% improvement
CSV with 1/3 quotes: 16297.541500 ms 2.539854% improvement
x86 NARROW master without inline (git revert
dc592a41557b072178f1798700bf9c69cd8e4235) default_toast_compression = pglz
TXT : 26243.084500 ms
CSV : 27719.564000 ms
TXT with 1/3 escapes: 29578.192750 ms
CSV with 1/3 quotes: 34467.571250 ms
x86 NARROW master with inline with v11patch default_toast_compression = pglz
TXT : 26371.996750 ms -0.491224% regression
CSV : 26137.186500 ms 5.708522% improvement
TXT with 1/3 escapes: 28080.201000 ms 5.064514% improvement
CSV with 1/3 quotes: 32557.377500 ms 5.542003% improvement
x86 WIDE master without inline (git revert
dc592a41557b072178f1798700bf9c69cd8e4235) default_toast_compression = pglz
TXT : 28734.774750 ms
CSV : 35700.485000 ms
TXT with 1/3 escapes: 32376.878250 ms
CSV with 1/3 quotes: 47024.985750 ms
x86 WIDE master with inline with v11patch default_toast_compression = pglz
TXT : 22753.755750 ms 20.814567% improvement
CSV : 22977.195500 ms 35.638982% improvement
TXT with 1/3 escapes: 29526.887000 ms 8.802551% improvement
CSV with 1/3 quotes: 40298.196750 ms 14.304712% improvement
--
-- Manni Wood EDB: https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 22:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-10 11:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-21 20:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Neil Conway <neil.conway@gmail.com>
2026-01-31 16:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 13:51 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 18:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 21:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-06 22:19 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 22:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-11 13:27 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-11 22:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-13 11:45 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-13 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 13:38 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-18 21:26 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 21:54 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 04:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-19 09:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 22:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-20 00:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-20 09:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-20 18:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-23 09:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 04:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-24 13:57 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 17:48 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-25 14:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-02 19:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-04 15:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-09 18:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-10 02:30 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
@ 2026-03-10 11:42 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
0 siblings, 0 replies; 178+ messages in thread
From: Nazir Bilal Yavuz @ 2026-03-10 11:42 UTC (permalink / raw)
To: Manni Wood <manni.wood@enterprisedb.com>; +Cc: Nathan Bossart <nathandbossart@gmail.com>; KAZAR Ayoub <ma_kazar@esi.dz>; Neil Conway <neil.conway@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
Hi,
On Tue, 10 Mar 2026 at 05:30, Manni Wood <manni.wood@enterprisedb.com> wrote:
>
> Here are some benchmarks showing what performance will look like for users who continue to use default_toast_compression = pglz.
>
> all compiled by meson with debugoptimized (-g -O2)
>
> arm NARROW master without inline (git revert dc592a41557b072178f1798700bf9c69cd8e4235) default_toast_compression = pglz
> TXT : 10055.141000 ms
> CSV : 10549.174500 ms
> TXT with 1/3 escapes: 10213.864750 ms
> CSV with 1/3 quotes: 12188.039000 ms
>
> arm NARROW master with inline with v11patch default_toast_compression = pglz
> TXT : 10070.153750 ms -0.149304% regression
> CSV : 10161.348750 ms 3.676361% improvement
> TXT with 1/3 escapes: 10618.005000 ms -3.956781% regression
> CSV with 1/3 quotes: 12279.366250 ms -0.749319% regression
>
> arm WIDE master without inline (git revert dc592a41557b072178f1798700bf9c69cd8e4235) default_toast_compression = pglz
> TXT : 11355.602750 ms
> CSV : 13893.110500 ms
> TXT with 1/3 escapes: 12872.690500 ms
> CSV with 1/3 quotes: 16722.262500 ms
>
> arm WIDE master with inline with v11patch default_toast_compression = pglz
> TXT : 9001.007250 ms 20.735099% improvement
> CSV : 8988.679750 ms 35.301171% improvement
> TXT with 1/3 escapes: 12191.137000 ms 5.294569% improvement
> CSV with 1/3 quotes: 16297.541500 ms 2.539854% improvement
>
>
> x86 NARROW master without inline (git revert dc592a41557b072178f1798700bf9c69cd8e4235) default_toast_compression = pglz
> TXT : 26243.084500 ms
> CSV : 27719.564000 ms
> TXT with 1/3 escapes: 29578.192750 ms
> CSV with 1/3 quotes: 34467.571250 ms
>
> x86 NARROW master with inline with v11patch default_toast_compression = pglz
> TXT : 26371.996750 ms -0.491224% regression
> CSV : 26137.186500 ms 5.708522% improvement
> TXT with 1/3 escapes: 28080.201000 ms 5.064514% improvement
> CSV with 1/3 quotes: 32557.377500 ms 5.542003% improvement
>
> x86 WIDE master without inline (git revert dc592a41557b072178f1798700bf9c69cd8e4235) default_toast_compression = pglz
> TXT : 28734.774750 ms
> CSV : 35700.485000 ms
> TXT with 1/3 escapes: 32376.878250 ms
> CSV with 1/3 quotes: 47024.985750 ms
>
> x86 WIDE master with inline with v11patch default_toast_compression = pglz
> TXT : 22753.755750 ms 20.814567% improvement
> CSV : 22977.195500 ms 35.638982% improvement
> TXT with 1/3 escapes: 29526.887000 ms 8.802551% improvement
> CSV with 1/3 quotes: 40298.196750 ms 14.304712% improvement
Thank you for the benchmark, results look nice! So, there is almost no
regression for both pglz and lz4 toast compression modes. Best case is
~60% improvement for the lz4 and ~35% improvement for the pglz.
--
Regards,
Nazir Bilal Yavuz
Microsoft
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 22:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-10 11:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-21 20:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Neil Conway <neil.conway@gmail.com>
2026-01-31 16:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 13:51 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 18:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 21:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-06 22:19 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 22:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-11 13:27 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-11 22:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-13 11:45 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-13 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 13:38 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-18 21:26 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 21:54 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 04:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-19 09:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 22:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-20 00:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-20 09:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-20 18:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-23 09:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 04:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-24 13:57 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 17:48 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-25 14:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-02 19:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-04 15:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-09 18:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
@ 2026-03-10 12:35 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-10 17:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
1 sibling, 1 reply; 178+ messages in thread
From: Nazir Bilal Yavuz @ 2026-03-10 12:35 UTC (permalink / raw)
To: Nathan Bossart <nathandbossart@gmail.com>; +Cc: Manni Wood <manni.wood@enterprisedb.com>; KAZAR Ayoub <ma_kazar@esi.dz>; Neil Conway <neil.conway@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
Hi,
On Mon, 9 Mar 2026 at 21:25, Nathan Bossart <nathandbossart@gmail.com> wrote:
>
> On Wed, Mar 04, 2026 at 06:15:53PM +0300, Nazir Bilal Yavuz wrote:
> > +#ifndef USE_NO_SIMD
> > +static bool CopyReadLineTextSIMDHelper(CopyFromState cstate, bool is_csv,
> > + bool *temp_hit_eof, int *temp_input_buf_ptr);
> > +#endif
>
> Should we inline this, too?
I think there is no need to inline this function. In the previous
version, SIMD code was in the main for loop which loops for every
character in the data. This means there was branching for every
character in the data. In the current version, SIMD code is outside of
this loop so there is no branching.
> > + /*
> > + * Do not disable SIMD when we hit EOL or EOF characters. In
> > + * practice, it does not matter for EOF because parsing ends
> > + * there, but we keep the behavior consistent.
> > + */
> > + if (!(simd_hit_eof || simd_hit_eol))
> > + cstate->simd_enabled = false;
>
> nitpick: I would personally avoid disabling it for EOF. It probably
> doesn't amount to much, but I don't see any point in the extra
> complexity/work solely for consistency.
Done. I thought that was a small change but this removed more
complexity than I thought.
>
> > + /*
> > + * We encountered a EOL or EOF on the first vector. This means
> > + * lines are not long enough to skip fully sized vector. If
> > + * this happens two times consecutively, then disable the
> > + * SIMD.
> > + */
> > + if (first_vector)
> > + {
> > + if (cstate->simd_failed_first_vector)
> > + cstate->simd_enabled = false;
> > +
> > + cstate->simd_failed_first_vector = true;
> > + }
>
> The first time I saw this, my mind immediately went to the extreme case
> where this likely regresses: alternating long and short lines. We might
> just want to disable it the first time we see a short line, like we do for
> special characters. This is another thing that we can improve
> independently later on.
I agree with you, done.
>
> > + /* First try to run SIMD, then continue with the scalar path */
> > + if (cstate->simd_enabled)
> > + {
> > + int temp_input_buf_ptr = input_buf_ptr;
> > + bool temp_hit_eof = false;
> > +
> > + result = CopyReadLineTextSIMDHelper(cstate, is_csv, &temp_hit_eof,
> > + &temp_input_buf_ptr);
> > + input_buf_ptr = temp_input_buf_ptr;
> > + hit_eof = temp_hit_eof;
>
> Given CopyReadLineTextSIMDHelper() doesn't have too much duplicated code,
> moving the SIMD stuff to its own function is nice. The temp variables seem
> a bit too magical to me, though. If those really make a difference, IMHO
> there ought to be a big comment explaining why.
I added a comment, please let me know if you wouldn't like it.
--
Regards,
Nazir Bilal Yavuz
Microsoft
Attachments:
[text/x-patch] v12-0001-Speed-up-COPY-FROM-text-CSV-parsing-using-SIMD.patch (10.0K, ../../CAN55FZ08kqmA+B9pzPDy-QstxAd=cK-RqjbR3cWBjPF_8-FXAw@mail.gmail.com/2-v12-0001-Speed-up-COPY-FROM-text-CSV-parsing-using-SIMD.patch)
download | inline diff:
From de695aaf5c7ceeb4f62d2352fabbb111047a4434 Mon Sep 17 00:00:00 2001
From: Nazir Bilal Yavuz <byavuz81@gmail.com>
Date: Wed, 4 Mar 2026 17:28:54 +0300
Subject: [PATCH v12] Speed up COPY FROM text/CSV parsing using SIMD
COPY FROM text and CSV parsing previously scanned the input buffer
one byte at a time looking for special characters (newline, carriage
return, backslash, and in CSV mode, quote and escape characters).
This patch adds a SIMD-accelerated fast path that processes a full
vector-width chunk per iteration, significantly reducing the number
of iterations needed on inputs with long lines.
A new helper function, CopyReadLineTextSIMDHelper(), loads chunks
of the input buffer into vector registers and checks for any special
characters using SIMD comparisons. When no special characters are
found, the entire chunk is skipped at once. When a special character
is found, the helper advances to that position and hands off to the
existing scalar code to handle it correctly.
To avoid a regression on inputs that contain many special characters,
SIMD is disabled for the remainder of the current input once a
non-EOL special character is encountered. SIMD is also disabled when
the processed line is shorter than a full vector, since SIMD can't
provide a benefit there.
Author: Shinya Kato <shinya11.kato@gmail.com>
Author: Nazir Bilal Yavuz <byavuz81@gmail.com>
Reviewed-by: Kazar Ayoub <ma_kazar@esi.dz>
Reviewed-by: Nathan Bossart <nathandbossart@gmail.com>
Reviewed-by: Neil Conway <neil.conway@gmail.com>
Reviewed-by: Andrew Dunstan <andrew@dunslane.net>
Reviewed-by: Manni Wood <manni.wood@enterprisedb.com>
Reviewed-by: Mark Wong <markwkm@gmail.com>
Discussion: https://postgr.es/m/CAOzEurSW8cNr6TPKsjrstnPfhf4QyQqB4tnPXGGe8N4e_v7Jig%40mail.gmail.com
---
src/backend/commands/copyfrom.c | 3 +
src/backend/commands/copyfromparse.c | 206 ++++++++++++++++++++++-
src/include/commands/copyfrom_internal.h | 3 +
3 files changed, 205 insertions(+), 7 deletions(-)
diff --git a/src/backend/commands/copyfrom.c b/src/backend/commands/copyfrom.c
index 2f42f55e229..fe18bd70890 100644
--- a/src/backend/commands/copyfrom.c
+++ b/src/backend/commands/copyfrom.c
@@ -1747,6 +1747,9 @@ BeginCopyFrom(ParseState *pstate,
cstate->cur_attval = NULL;
cstate->relname_only = false;
+ /* Initialize SIMD */
+ cstate->simd_enabled = true;
+
/*
* Allocate buffers for the input pipeline.
*
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index 84c8809a889..166b1c4c415 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -72,6 +72,7 @@
#include "miscadmin.h"
#include "pgstat.h"
#include "port/pg_bswap.h"
+#include "port/simd.h"
#include "utils/builtins.h"
#include "utils/rel.h"
#include "utils/wait_event.h"
@@ -159,6 +160,12 @@ static pg_attribute_always_inline bool NextCopyFromRawFieldsInternal(CopyFromSta
int *nfields,
bool is_csv);
+/* SIMD functions */
+#ifndef USE_NO_SIMD
+static bool CopyReadLineTextSIMDHelper(CopyFromState cstate, bool is_csv,
+ bool *temp_hit_eof, int *temp_input_buf_ptr);
+#endif
+
/* Low-level communications functions */
static int CopyGetData(CopyFromState cstate, void *databuf,
@@ -1311,6 +1318,155 @@ CopyReadLine(CopyFromState cstate, bool is_csv)
return result;
}
+#ifndef USE_NO_SIMD
+/*
+ * Use SIMD instructions to efficiently scan the input buffer for special
+ * characters (e.g., newline, carriage return, quote, and escape). This is
+ * faster than byte-by-byte iteration, especially on large buffers.
+ *
+ * Note that, SIMD may become slower when the input contains many special
+ * characters. To avoid this regression, we disable SIMD for the rest of the
+ * input once we encounter a special character which isn't EOL.
+ * Also, SIMD is disabled when it encounters a short line that SIMD can't
+ * create a full sized Vector, too.
+ */
+static bool
+CopyReadLineTextSIMDHelper(CopyFromState cstate, bool is_csv, bool *temp_hit_eof, int *temp_input_buf_ptr)
+{
+ char quotec = '\0';
+ char escapec = '\0';
+ char *copy_input_buf;
+ int input_buf_ptr;
+ int copy_buf_len;
+ bool result = false;
+ bool unique_escapec = false;
+ bool first_vector = true;
+ Vector8 nl = vector8_broadcast('\n');
+ Vector8 cr = vector8_broadcast('\r');
+ Vector8 bs = vector8_broadcast('\\');
+ Vector8 quote = vector8_broadcast(0);
+ Vector8 escape = vector8_broadcast(0);
+
+ if (is_csv)
+ {
+ quotec = cstate->opts.quote[0];
+ escapec = cstate->opts.escape[0];
+
+ quote = vector8_broadcast(quotec);
+ if (quotec != escapec)
+ {
+ unique_escapec = true;
+ escape = vector8_broadcast(escapec);
+ }
+ }
+
+ /* For a little extra speed we copy these into local variables */
+ copy_input_buf = cstate->input_buf;
+ input_buf_ptr = cstate->input_buf_index;
+ copy_buf_len = cstate->input_buf_len;
+
+ while (true)
+ {
+ /* Load more data if needed */
+ if (sizeof(Vector8) > copy_buf_len - input_buf_ptr)
+ {
+ REFILL_LINEBUF;
+
+ CopyLoadInputBuf(cstate);
+ /* update our local variables */
+ *temp_hit_eof = cstate->input_reached_eof;
+ input_buf_ptr = cstate->input_buf_index;
+ copy_buf_len = cstate->input_buf_len;
+
+ /*
+ * If we are completely out of data, break out of the loop,
+ * reporting EOF.
+ */
+ if (INPUT_BUF_BYTES(cstate) <= 0)
+ {
+ result = true;
+ break;
+ }
+ }
+
+ if (copy_buf_len - input_buf_ptr >= sizeof(Vector8))
+ {
+ Vector8 chunk;
+ Vector8 match = vector8_broadcast(0);
+
+ /* Load a chunk of data into a vector register */
+ vector8_load(&chunk, (const uint8 *) ©_input_buf[input_buf_ptr]);
+
+ if (is_csv)
+ {
+ match = vector8_or(vector8_eq(chunk, nl), vector8_eq(chunk, cr));
+ match = vector8_or(match, vector8_eq(chunk, quote));
+ if (unique_escapec)
+ match = vector8_or(match, vector8_eq(chunk, escape));
+ }
+ else
+ {
+ match = vector8_or(vector8_eq(chunk, nl), vector8_eq(chunk, cr));
+ match = vector8_or(match, vector8_eq(chunk, bs));
+ }
+
+ /* Check if we found any special characters */
+ if (vector8_is_highbit_set(match))
+ {
+ /*
+ * Found a special character. Advance up to that point and let
+ * the scalar code handle it.
+ */
+ uint32 mask;
+ int advance;
+ char c;
+
+ mask = vector8_highbit_mask(match);
+ advance = pg_rightmost_one_pos32(mask);
+
+ input_buf_ptr += advance;
+ c = copy_input_buf[input_buf_ptr];
+
+ /*
+ * We encountered a special character in the first vector.
+ * This means line is not long enough to skip fully sized
+ * vector. To be cautios, disable SIMD for the rest.
+ *
+ * Otherwise, do not disable SIMD when we hit EOL characters.
+ * We don't check for EOF because parsing ends there.
+ */
+ if (first_vector || !(c == '\r' || c == '\n'))
+ cstate->simd_enabled = false;
+
+ break;
+ }
+ else
+ {
+ /* No special characters found, so skip the entire chunk */
+ input_buf_ptr += sizeof(Vector8);
+ first_vector = false;
+ }
+ }
+
+ /*
+ * Although we refill linebuf, there is not enough character to fill
+ * full sized vector. This doesn't mean that we encountered a line
+ * that is not enough to fill a full sized vector.
+ *
+ * Scalar code will handle the rest for this line. Then, SIMD will
+ * continue from the next line.
+ */
+ else
+ {
+ break;
+ }
+ }
+
+ *temp_input_buf_ptr = input_buf_ptr;
+ return result;
+}
+#endif
+
/*
* CopyReadLineText - inner loop of CopyReadLine for text mode
*/
@@ -1339,6 +1495,49 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
escapec = '\0';
}
+ /*
+ * input_buf_ptr might be updated in the SIMD Helper function, so it needs
+ * to be set before calling CopyReadLineTextSIMDHelper().
+ */
+ input_buf_ptr = cstate->input_buf_index;
+
+#ifndef USE_NO_SIMD
+ /* First try to run SIMD, then continue with the scalar path */
+ if (cstate->simd_enabled)
+ {
+ /*
+ * Temporary variables are used here instead of passing the actual
+ * variables (especially input_buf_ptr) directly to the helper. Taking
+ * the address of a local variable might force the compiler to
+ * allocate it on the stack rather than in a register. Because
+ * input_buf_ptr is used heavily in the hot scalar path below, keeping
+ * it in a register is important for performance.
+ */
+ int temp_input_buf_ptr;
+ bool temp_hit_eof = hit_eof;
+
+ result = CopyReadLineTextSIMDHelper(cstate, is_csv, &temp_hit_eof,
+ &temp_input_buf_ptr);
+ input_buf_ptr = temp_input_buf_ptr;
+ hit_eof = temp_hit_eof;
+
+ /* Short exit from SIMD */
+ if (result)
+ {
+ /*
+ * Transfer any still-uncopied data to line_buf.
+ */
+ REFILL_LINEBUF;
+
+ return result;
+ }
+ }
+#endif
+
+ /* For a little extra speed we copy these into local variables */
+ copy_input_buf = cstate->input_buf;
+ copy_buf_len = cstate->input_buf_len;
+
/*
* The objective of this loop is to transfer the entire next input line
* into line_buf. Hence, we only care for detecting newlines (\r and/or
@@ -1360,14 +1559,7 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
* character to examine; any characters from input_buf_index to
* input_buf_ptr have been determined to be part of the line, but not yet
* transferred to line_buf.
- *
- * For a little extra speed within the loop, we copy input_buf and
- * input_buf_len into local variables.
*/
- copy_input_buf = cstate->input_buf;
- input_buf_ptr = cstate->input_buf_index;
- copy_buf_len = cstate->input_buf_len;
-
for (;;)
{
int prev_raw_ptr;
diff --git a/src/include/commands/copyfrom_internal.h b/src/include/commands/copyfrom_internal.h
index f892c343157..5b020bf4d0b 100644
--- a/src/include/commands/copyfrom_internal.h
+++ b/src/include/commands/copyfrom_internal.h
@@ -89,6 +89,9 @@ typedef struct CopyFromStateData
const char *cur_attval; /* current att value for error messages */
bool relname_only; /* don't output line number, att, etc. */
+ /* SIMD variables */
+ bool simd_enabled;
+
/*
* Working state
*/
--
2.47.3
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 22:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-10 11:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-21 20:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Neil Conway <neil.conway@gmail.com>
2026-01-31 16:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 13:51 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 18:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 21:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-06 22:19 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 22:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-11 13:27 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-11 22:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-13 11:45 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-13 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 13:38 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-18 21:26 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 21:54 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 04:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-19 09:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 22:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-20 00:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-20 09:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-20 18:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-23 09:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 04:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-24 13:57 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 17:48 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-25 14:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-02 19:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-04 15:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-09 18:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-10 12:35 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
@ 2026-03-10 17:10 ` Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 11:36 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
0 siblings, 1 reply; 178+ messages in thread
From: Nathan Bossart @ 2026-03-10 17:10 UTC (permalink / raw)
To: Nazir Bilal Yavuz <byavuz81@gmail.com>; +Cc: Manni Wood <manni.wood@enterprisedb.com>; KAZAR Ayoub <ma_kazar@esi.dz>; Neil Conway <neil.conway@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
On Tue, Mar 10, 2026 at 03:35:30PM +0300, Nazir Bilal Yavuz wrote:
> Subject: [PATCH v12] Speed up COPY FROM text/CSV parsing using SIMD
This looks pretty good to me. I'm hoping to take a closer look in the near
future, but I think we are approaching something committable.
--
nathan
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 22:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-10 11:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-21 20:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Neil Conway <neil.conway@gmail.com>
2026-01-31 16:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 13:51 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 18:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 21:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-06 22:19 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 22:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-11 13:27 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-11 22:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-13 11:45 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-13 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 13:38 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-18 21:26 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 21:54 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 04:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-19 09:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 22:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-20 00:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-20 09:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-20 18:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-23 09:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 04:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-24 13:57 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 17:48 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-25 14:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-02 19:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-04 15:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-09 18:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-10 12:35 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-10 17:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
@ 2026-03-11 11:36 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 12:19 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-03-11 18:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
0 siblings, 2 replies; 178+ messages in thread
From: Nazir Bilal Yavuz @ 2026-03-11 11:36 UTC (permalink / raw)
To: Nathan Bossart <nathandbossart@gmail.com>; +Cc: Manni Wood <manni.wood@enterprisedb.com>; KAZAR Ayoub <ma_kazar@esi.dz>; Neil Conway <neil.conway@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
Hi,
On Tue, 10 Mar 2026 at 20:10, Nathan Bossart <nathandbossart@gmail.com> wrote:
>
> On Tue, Mar 10, 2026 at 03:35:30PM +0300, Nazir Bilal Yavuz wrote:
> > Subject: [PATCH v12] Speed up COPY FROM text/CSV parsing using SIMD
>
> This looks pretty good to me. I'm hoping to take a closer look in the near
> future, but I think we are approaching something committable.
Thanks for looking into it!
I am attaching v13 of the patch.
0001 is basically some typo fixes on top of v12, no functional changes.
0002 has an attempt to remove some branches from SIMD code but since
it is kind of functional change, I wanted to attach that as another
patch. I think we can apply some parts of this, if not all.
--
Regards,
Nazir Bilal Yavuz
Microsoft
Attachments:
[text/x-patch] v13-0001-Speed-up-COPY-FROM-text-CSV-parsing-using-SIMD.patch (10.0K, ../../CAN55FZ3jXs7XDsP_-v_jUBquRu4uAdheN3xcmW=WhAyKwFLSjg@mail.gmail.com/2-v13-0001-Speed-up-COPY-FROM-text-CSV-parsing-using-SIMD.patch)
download | inline diff:
From f3e9a234ddc537544d510ad344eb1b8eb2127855 Mon Sep 17 00:00:00 2001
From: Nazir Bilal Yavuz <byavuz81@gmail.com>
Date: Wed, 11 Mar 2026 14:04:29 +0300
Subject: [PATCH v13 1/2] Speed up COPY FROM text/CSV parsing using SIMD
COPY FROM text and CSV parsing previously scanned the input buffer
one byte at a time looking for special characters (newline, carriage
return, backslash, and in CSV mode, quote and escape characters).
This patch adds a SIMD-accelerated fast path that processes a full
vector-width chunk per iteration, significantly reducing the number
of iterations needed on inputs with long lines.
A new helper function, CopyReadLineTextSIMDHelper(), loads chunks
of the input buffer into vector registers and checks for any special
characters using SIMD comparisons. When no special characters are
found, the entire chunk is skipped at once. When a special character
is found, the helper advances to that position and hands off to the
existing scalar code to handle it correctly.
To avoid a regression on inputs that contain many special characters,
SIMD is disabled for the remainder of the current input once a
non-EOL special character is encountered. SIMD is also disabled when
the processed line is shorter than a full vector, since SIMD can't
provide a benefit there.
Author: Shinya Kato <shinya11.kato@gmail.com>
Author: Nazir Bilal Yavuz <byavuz81@gmail.com>
Reviewed-by: Kazar Ayoub <ma_kazar@esi.dz>
Reviewed-by: Nathan Bossart <nathandbossart@gmail.com>
Reviewed-by: Neil Conway <neil.conway@gmail.com>
Reviewed-by: Andrew Dunstan <andrew@dunslane.net>
Reviewed-by: Manni Wood <manni.wood@enterprisedb.com>
Reviewed-by: Mark Wong <markwkm@gmail.com>
Discussion: https://postgr.es/m/CAOzEurSW8cNr6TPKsjrstnPfhf4QyQqB4tnPXGGe8N4e_v7Jig%40mail.gmail.com
---
src/backend/commands/copyfrom.c | 3 +
src/backend/commands/copyfromparse.c | 206 ++++++++++++++++++++++-
src/include/commands/copyfrom_internal.h | 3 +
3 files changed, 205 insertions(+), 7 deletions(-)
diff --git a/src/backend/commands/copyfrom.c b/src/backend/commands/copyfrom.c
index 2f42f55e229..fe18bd70890 100644
--- a/src/backend/commands/copyfrom.c
+++ b/src/backend/commands/copyfrom.c
@@ -1747,6 +1747,9 @@ BeginCopyFrom(ParseState *pstate,
cstate->cur_attval = NULL;
cstate->relname_only = false;
+ /* Initialize SIMD */
+ cstate->simd_enabled = true;
+
/*
* Allocate buffers for the input pipeline.
*
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index 84c8809a889..9f1256353c4 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -72,6 +72,7 @@
#include "miscadmin.h"
#include "pgstat.h"
#include "port/pg_bswap.h"
+#include "port/simd.h"
#include "utils/builtins.h"
#include "utils/rel.h"
#include "utils/wait_event.h"
@@ -159,6 +160,12 @@ static pg_attribute_always_inline bool NextCopyFromRawFieldsInternal(CopyFromSta
int *nfields,
bool is_csv);
+/* SIMD functions */
+#ifndef USE_NO_SIMD
+static bool CopyReadLineTextSIMDHelper(CopyFromState cstate, bool is_csv,
+ bool *temp_hit_eof, int *temp_input_buf_ptr);
+#endif
+
/* Low-level communications functions */
static int CopyGetData(CopyFromState cstate, void *databuf,
@@ -1311,6 +1318,155 @@ CopyReadLine(CopyFromState cstate, bool is_csv)
return result;
}
+#ifndef USE_NO_SIMD
+/*
+ * Use SIMD instructions to efficiently scan the input buffer for special
+ * characters (e.g., newline, carriage return, quote, and escape). This is
+ * faster than byte-by-byte iteration, especially on large buffers.
+ *
+ * Note that, SIMD may become slower when the input contains many special
+ * characters. To avoid this regression, we disable SIMD for the rest of the
+ * input once we encounter a special character which isn't EOL.
+ * Also, SIMD is disabled when it encounters a short line that SIMD can't
+ * create a full sized vector, too.
+ */
+static bool
+CopyReadLineTextSIMDHelper(CopyFromState cstate, bool is_csv, bool *temp_hit_eof, int *temp_input_buf_ptr)
+{
+ char quotec = '\0';
+ char escapec = '\0';
+ char *copy_input_buf;
+ int input_buf_ptr;
+ int copy_buf_len;
+ bool result = false;
+ bool unique_escapec = false;
+ bool first_vector = true;
+ Vector8 nl = vector8_broadcast('\n');
+ Vector8 cr = vector8_broadcast('\r');
+ Vector8 bs = vector8_broadcast('\\');
+ Vector8 quote = vector8_broadcast(0);
+ Vector8 escape = vector8_broadcast(0);
+
+ if (is_csv)
+ {
+ quotec = cstate->opts.quote[0];
+ escapec = cstate->opts.escape[0];
+
+ quote = vector8_broadcast(quotec);
+ if (quotec != escapec)
+ {
+ unique_escapec = true;
+ escape = vector8_broadcast(escapec);
+ }
+ }
+
+ /* For a little extra speed we copy these into local variables */
+ copy_input_buf = cstate->input_buf;
+ input_buf_ptr = cstate->input_buf_index;
+ copy_buf_len = cstate->input_buf_len;
+
+ while (true)
+ {
+ /* Load more data if needed */
+ if (sizeof(Vector8) > copy_buf_len - input_buf_ptr)
+ {
+ REFILL_LINEBUF;
+
+ CopyLoadInputBuf(cstate);
+ /* update our local variables */
+ *temp_hit_eof = cstate->input_reached_eof;
+ input_buf_ptr = cstate->input_buf_index;
+ copy_buf_len = cstate->input_buf_len;
+
+ /*
+ * If we are completely out of data, break out of the loop,
+ * reporting EOF.
+ */
+ if (INPUT_BUF_BYTES(cstate) <= 0)
+ {
+ result = true;
+ break;
+ }
+ }
+
+ if (copy_buf_len - input_buf_ptr >= sizeof(Vector8))
+ {
+ Vector8 chunk;
+ Vector8 match = vector8_broadcast(0);
+
+ /* Load a chunk of data into a vector register */
+ vector8_load(&chunk, (const uint8 *) ©_input_buf[input_buf_ptr]);
+
+ if (is_csv)
+ {
+ match = vector8_or(vector8_eq(chunk, nl), vector8_eq(chunk, cr));
+ match = vector8_or(match, vector8_eq(chunk, quote));
+ if (unique_escapec)
+ match = vector8_or(match, vector8_eq(chunk, escape));
+ }
+ else
+ {
+ match = vector8_or(vector8_eq(chunk, nl), vector8_eq(chunk, cr));
+ match = vector8_or(match, vector8_eq(chunk, bs));
+ }
+
+ /* Check if we found any special characters */
+ if (vector8_is_highbit_set(match))
+ {
+ /*
+ * Found a special character. Advance up to that point and let
+ * the scalar code handle it.
+ */
+ uint32 mask;
+ int advance;
+ char c;
+
+ mask = vector8_highbit_mask(match);
+ advance = pg_rightmost_one_pos32(mask);
+
+ input_buf_ptr += advance;
+ c = copy_input_buf[input_buf_ptr];
+
+ /*
+ * If we encountered a special character in the first vector,
+ * this means line is not long enough to skip fully sized
+ * vector. To be cautious, disable SIMD for the rest.
+ *
+ * Otherwise, do not disable SIMD when we hit EOL characters.
+ * We don't check for EOF because parsing ends there.
+ */
+ if (first_vector || !(c == '\r' || c == '\n'))
+ cstate->simd_enabled = false;
+
+ break;
+ }
+ else
+ {
+ /* No special characters found, so skip the entire chunk */
+ input_buf_ptr += sizeof(Vector8);
+ first_vector = false;
+ }
+ }
+
+ /*
+ * Although we refill linebuf, there is not enough character to fill
+ * full sized vector. This doesn't mean that we encountered a line
+ * that is not enough to fill a full sized vector.
+ *
+ * Scalar code will handle the rest for this line. Then, SIMD will
+ * continue from the next line.
+ */
+ else
+ {
+ break;
+ }
+ }
+
+ *temp_input_buf_ptr = input_buf_ptr;
+ return result;
+}
+#endif
+
/*
* CopyReadLineText - inner loop of CopyReadLine for text mode
*/
@@ -1339,6 +1495,49 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
escapec = '\0';
}
+ /*
+ * input_buf_ptr might be updated in the SIMD Helper function, so it needs
+ * to be set before calling CopyReadLineTextSIMDHelper().
+ */
+ input_buf_ptr = cstate->input_buf_index;
+
+#ifndef USE_NO_SIMD
+ /* First try to run SIMD, then continue with the scalar path */
+ if (cstate->simd_enabled)
+ {
+ /*
+ * Temporary variables are used here instead of passing the actual
+ * variables (especially input_buf_ptr) directly to the helper. Taking
+ * the address of a local variable might force the compiler to
+ * allocate it on the stack rather than in a register. Because
+ * input_buf_ptr is used heavily in the hot scalar path below, keeping
+ * it in a register is important for performance.
+ */
+ int temp_input_buf_ptr;
+ bool temp_hit_eof = hit_eof;
+
+ result = CopyReadLineTextSIMDHelper(cstate, is_csv, &temp_hit_eof,
+ &temp_input_buf_ptr);
+ input_buf_ptr = temp_input_buf_ptr;
+ hit_eof = temp_hit_eof;
+
+ /* Early exit from SIMD */
+ if (result)
+ {
+ /*
+ * Transfer any still-uncopied data to line_buf.
+ */
+ REFILL_LINEBUF;
+
+ return result;
+ }
+ }
+#endif
+
+ /* For a little extra speed we copy these into local variables */
+ copy_input_buf = cstate->input_buf;
+ copy_buf_len = cstate->input_buf_len;
+
/*
* The objective of this loop is to transfer the entire next input line
* into line_buf. Hence, we only care for detecting newlines (\r and/or
@@ -1360,14 +1559,7 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
* character to examine; any characters from input_buf_index to
* input_buf_ptr have been determined to be part of the line, but not yet
* transferred to line_buf.
- *
- * For a little extra speed within the loop, we copy input_buf and
- * input_buf_len into local variables.
*/
- copy_input_buf = cstate->input_buf;
- input_buf_ptr = cstate->input_buf_index;
- copy_buf_len = cstate->input_buf_len;
-
for (;;)
{
int prev_raw_ptr;
diff --git a/src/include/commands/copyfrom_internal.h b/src/include/commands/copyfrom_internal.h
index f892c343157..5b020bf4d0b 100644
--- a/src/include/commands/copyfrom_internal.h
+++ b/src/include/commands/copyfrom_internal.h
@@ -89,6 +89,9 @@ typedef struct CopyFromStateData
const char *cur_attval; /* current att value for error messages */
bool relname_only; /* don't output line number, att, etc. */
+ /* SIMD variables */
+ bool simd_enabled;
+
/*
* Working state
*/
--
2.47.3
[text/x-patch] v13-0002-Upcoming-improvements.patch (2.4K, ../../CAN55FZ3jXs7XDsP_-v_jUBquRu4uAdheN3xcmW=WhAyKwFLSjg@mail.gmail.com/3-v13-0002-Upcoming-improvements.patch)
download | inline diff:
From 1006dac44cb208a8b164f2553772de942e79c2d4 Mon Sep 17 00:00:00 2001
From: Nazir Bilal Yavuz <byavuz81@gmail.com>
Date: Wed, 11 Mar 2026 14:04:55 +0300
Subject: [PATCH v13 2/2] Upcoming improvements
---
src/backend/commands/copyfromparse.c | 27 ++++++++-------------------
1 file changed, 8 insertions(+), 19 deletions(-)
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index 9f1256353c4..55159b0122c 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -1333,8 +1333,6 @@ CopyReadLine(CopyFromState cstate, bool is_csv)
static bool
CopyReadLineTextSIMDHelper(CopyFromState cstate, bool is_csv, bool *temp_hit_eof, int *temp_input_buf_ptr)
{
- char quotec = '\0';
- char escapec = '\0';
char *copy_input_buf;
int input_buf_ptr;
int copy_buf_len;
@@ -1343,16 +1341,15 @@ CopyReadLineTextSIMDHelper(CopyFromState cstate, bool is_csv, bool *temp_hit_eof
bool first_vector = true;
Vector8 nl = vector8_broadcast('\n');
Vector8 cr = vector8_broadcast('\r');
- Vector8 bs = vector8_broadcast('\\');
- Vector8 quote = vector8_broadcast(0);
+ Vector8 bs_or_quote = vector8_broadcast('\\');
Vector8 escape = vector8_broadcast(0);
if (is_csv)
{
- quotec = cstate->opts.quote[0];
- escapec = cstate->opts.escape[0];
+ char quotec = cstate->opts.quote[0];
+ char escapec = cstate->opts.escape[0];
- quote = vector8_broadcast(quotec);
+ bs_or_quote = vector8_broadcast(quotec);
if (quotec != escapec)
{
unique_escapec = true;
@@ -1397,18 +1394,10 @@ CopyReadLineTextSIMDHelper(CopyFromState cstate, bool is_csv, bool *temp_hit_eof
/* Load a chunk of data into a vector register */
vector8_load(&chunk, (const uint8 *) ©_input_buf[input_buf_ptr]);
- if (is_csv)
- {
- match = vector8_or(vector8_eq(chunk, nl), vector8_eq(chunk, cr));
- match = vector8_or(match, vector8_eq(chunk, quote));
- if (unique_escapec)
- match = vector8_or(match, vector8_eq(chunk, escape));
- }
- else
- {
- match = vector8_or(vector8_eq(chunk, nl), vector8_eq(chunk, cr));
- match = vector8_or(match, vector8_eq(chunk, bs));
- }
+ match = vector8_or(vector8_eq(chunk, nl), vector8_eq(chunk, cr));
+ match = vector8_or(match, vector8_eq(chunk, bs_or_quote));
+ if (unique_escapec)
+ match = vector8_or(match, vector8_eq(chunk, escape));
/* Check if we found any special characters */
if (vector8_is_highbit_set(match))
--
2.47.3
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 22:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-10 11:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-21 20:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Neil Conway <neil.conway@gmail.com>
2026-01-31 16:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 13:51 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 18:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 21:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-06 22:19 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 22:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-11 13:27 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-11 22:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-13 11:45 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-13 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 13:38 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-18 21:26 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 21:54 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 04:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-19 09:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 22:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-20 00:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-20 09:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-20 18:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-23 09:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 04:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-24 13:57 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 17:48 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-25 14:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-02 19:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-04 15:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-09 18:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-10 12:35 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-10 17:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 11:36 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
@ 2026-03-11 12:19 ` KAZAR Ayoub <ma_kazar@esi.dz>
2026-03-11 13:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
1 sibling, 1 reply; 178+ messages in thread
From: KAZAR Ayoub @ 2026-03-11 12:19 UTC (permalink / raw)
To: Nazir Bilal Yavuz <byavuz81@gmail.com>; +Cc: Nathan Bossart <nathandbossart@gmail.com>; Manni Wood <manni.wood@enterprisedb.com>; Neil Conway <neil.conway@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
On Wed, Mar 11, 2026, 12:36 PM Nazir Bilal Yavuz <byavuz81@gmail.com> wrote:
> 0002 has an attempt to remove some branches from SIMD code but since
> it is kind of functional change, I wanted to attach that as another
> patch. I think we can apply some parts of this, if not all.
>
0002 sounds really good to have, haven't measured the diff but it's very
logical.
Another quick question though, do we need USE_NO_SIMD for any reason? I
just remembered that there's some simd paths like json that don't use it.
Regards,
Ayoub
>
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 22:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-10 11:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-21 20:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Neil Conway <neil.conway@gmail.com>
2026-01-31 16:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 13:51 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 18:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 21:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-06 22:19 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 22:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-11 13:27 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-11 22:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-13 11:45 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-13 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 13:38 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-18 21:26 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 21:54 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 04:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-19 09:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 22:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-20 00:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-20 09:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-20 18:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-23 09:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 04:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-24 13:57 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 17:48 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-25 14:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-02 19:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-04 15:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-09 18:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-10 12:35 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-10 17:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 11:36 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 12:19 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
@ 2026-03-11 13:10 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 13:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
0 siblings, 1 reply; 178+ messages in thread
From: Nazir Bilal Yavuz @ 2026-03-11 13:10 UTC (permalink / raw)
To: KAZAR Ayoub <ma_kazar@esi.dz>; +Cc: Nathan Bossart <nathandbossart@gmail.com>; Manni Wood <manni.wood@enterprisedb.com>; Neil Conway <neil.conway@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
Hi,
On Wed, 11 Mar 2026 at 15:19, KAZAR Ayoub <ma_kazar@esi.dz> wrote:
>
> On Wed, Mar 11, 2026, 12:36 PM Nazir Bilal Yavuz <byavuz81@gmail.com> wrote:
>>
>> 0002 has an attempt to remove some branches from SIMD code but since
>> it is kind of functional change, I wanted to attach that as another
>> patch. I think we can apply some parts of this, if not all.
>
> 0002 sounds really good to have, haven't measured the diff but it's very logical.
I agree with you. I saw very small speedups like 1%-2% but I think
changes make sense regardless of the performance improvement.
> Another quick question though, do we need USE_NO_SIMD for any reason? I just remembered that there's some simd paths like json that don't use it.
vector8_eq() and vector8_highbit_mask() don't have non-SIMD
implementations, so we need to use USE_NO_SIMD.
--
Regards,
Nazir Bilal Yavuz
Microsoft
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 22:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-10 11:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-21 20:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Neil Conway <neil.conway@gmail.com>
2026-01-31 16:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 13:51 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 18:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 21:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-06 22:19 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 22:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-11 13:27 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-11 22:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-13 11:45 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-13 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 13:38 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-18 21:26 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 21:54 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 04:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-19 09:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 22:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-20 00:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-20 09:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-20 18:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-23 09:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 04:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-24 13:57 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 17:48 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-25 14:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-02 19:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-04 15:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-09 18:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-10 12:35 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-10 17:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 11:36 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 12:19 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-03-11 13:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
@ 2026-03-11 13:23 ` KAZAR Ayoub <ma_kazar@esi.dz>
0 siblings, 0 replies; 178+ messages in thread
From: KAZAR Ayoub @ 2026-03-11 13:23 UTC (permalink / raw)
To: Nazir Bilal Yavuz <byavuz81@gmail.com>; +Cc: Nathan Bossart <nathandbossart@gmail.com>; Manni Wood <manni.wood@enterprisedb.com>; Neil Conway <neil.conway@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
On Wed, Mar 11, 2026 at 2:10 PM Nazir Bilal Yavuz <byavuz81@gmail.com>
wrote:
> Hi,
>
> On Wed, 11 Mar 2026 at 15:19, KAZAR Ayoub <ma_kazar@esi.dz> wrote:
> >
> > On Wed, Mar 11, 2026, 12:36 PM Nazir Bilal Yavuz <byavuz81@gmail.com>
> wrote:
> >>
> >> 0002 has an attempt to remove some branches from SIMD code but since
> >> it is kind of functional change, I wanted to attach that as another
> >> patch. I think we can apply some parts of this, if not all.
> >
> > 0002 sounds really good to have, haven't measured the diff but it's very
> logical.
>
> I agree with you. I saw very small speedups like 1%-2% but I think
> changes make sense regardless of the performance improvement.
>
> > Another quick question though, do we need USE_NO_SIMD for any reason? I
> just remembered that there's some simd paths like json that don't use it.
>
> vector8_eq() and vector8_highbit_mask() don't have non-SIMD
> implementations, so we need to use USE_NO_SIMD.
>
Aha ! that's true, thanks.
Regards,
Ayoub
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 22:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-10 11:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-21 20:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Neil Conway <neil.conway@gmail.com>
2026-01-31 16:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 13:51 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 18:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 21:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-06 22:19 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 22:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-11 13:27 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-11 22:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-13 11:45 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-13 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 13:38 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-18 21:26 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 21:54 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 04:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-19 09:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 22:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-20 00:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-20 09:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-20 18:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-23 09:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 04:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-24 13:57 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 17:48 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-25 14:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-02 19:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-04 15:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-09 18:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-10 12:35 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-10 17:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 11:36 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
@ 2026-03-11 18:09 ` Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 18:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
1 sibling, 1 reply; 178+ messages in thread
From: Nathan Bossart @ 2026-03-11 18:09 UTC (permalink / raw)
To: Nazir Bilal Yavuz <byavuz81@gmail.com>; +Cc: Manni Wood <manni.wood@enterprisedb.com>; KAZAR Ayoub <ma_kazar@esi.dz>; Neil Conway <neil.conway@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
On Wed, Mar 11, 2026 at 02:36:46PM +0300, Nazir Bilal Yavuz wrote:
> 0002 has an attempt to remove some branches from SIMD code but since
> it is kind of functional change, I wanted to attach that as another
> patch. I think we can apply some parts of this, if not all.
Could you describe what this is doing and what the performance impact is?
--
nathan
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 22:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-10 11:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-21 20:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Neil Conway <neil.conway@gmail.com>
2026-01-31 16:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 13:51 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 18:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 21:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-06 22:19 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 22:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-11 13:27 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-11 22:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-13 11:45 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-13 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 13:38 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-18 21:26 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 21:54 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 04:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-19 09:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 22:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-20 00:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-20 09:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-20 18:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-23 09:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 04:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-24 13:57 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 17:48 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-25 14:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-02 19:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-04 15:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-09 18:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-10 12:35 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-10 17:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 11:36 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 18:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
@ 2026-03-11 18:49 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 19:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
0 siblings, 1 reply; 178+ messages in thread
From: Nazir Bilal Yavuz @ 2026-03-11 18:49 UTC (permalink / raw)
To: Nathan Bossart <nathandbossart@gmail.com>; +Cc: Manni Wood <manni.wood@enterprisedb.com>; KAZAR Ayoub <ma_kazar@esi.dz>; Neil Conway <neil.conway@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
Hi,
On Wed, 11 Mar 2026 at 21:09, Nathan Bossart <nathandbossart@gmail.com> wrote:
>
> On Wed, Mar 11, 2026 at 02:36:46PM +0300, Nazir Bilal Yavuz wrote:
> > 0002 has an attempt to remove some branches from SIMD code but since
> > it is kind of functional change, I wanted to attach that as another
> > patch. I think we can apply some parts of this, if not all.
>
> Could you describe what this is doing and what the performance impact is?
SIMD code check these characters:
csv mode: nl, cr, quote and possibly escape.
text mode: nl, cr and bs.
v12 checks them like that:
if (is_csv)
{
match = vector8_or(vector8_eq(chunk, nl),
vector8_eq(chunk, cr));
match = vector8_or(match, vector8_eq(chunk, quote));
if (unique_escapec)
match = vector8_or(match, vector8_eq(chunk, escape));
}
else
{
match = vector8_or(vector8_eq(chunk, nl),
vector8_eq(chunk, cr));
match = vector8_or(match, vector8_eq(chunk, bs));
}
But actually we know that we will definitely check nl, cr and one of
the quote or bs characters in the code. So, we can introduce a new
variable named bs_or_quote, it will be equal to bs if the mode is text
and it will be equal to quote if the mode is csv. Then, we can remove
the 'if (is_csv)' check and only check for escape ('if
(unique_escapec)'). Now code will look like that:
match = vector8_or(vector8_eq(chunk, nl), vector8_eq(chunk, cr));
match = vector8_or(match, vector8_eq(chunk, bs_or_quote));
if (unique_escapec)
match = vector8_or(match, vector8_eq(chunk, escape));
That is what v13-0002 does. I saw 1%-2% speedups with this change and
there was no regression.
Regardless of introducing the bs_or_quote variable, we can move 'match
= vector8_or(vector8_eq(chunk, nl), vector8_eq(chunk, cr));' outside
of the if checks, though.
--
Regards,
Nazir Bilal Yavuz
Microsoft
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 22:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-10 11:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-21 20:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Neil Conway <neil.conway@gmail.com>
2026-01-31 16:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 13:51 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 18:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 21:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-06 22:19 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 22:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-11 13:27 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-11 22:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-13 11:45 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-13 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 13:38 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-18 21:26 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 21:54 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 04:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-19 09:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 22:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-20 00:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-20 09:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-20 18:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-23 09:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 04:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-24 13:57 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 17:48 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-25 14:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-02 19:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-04 15:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-09 18:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-10 12:35 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-10 17:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 11:36 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 18:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 18:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
@ 2026-03-11 19:02 ` Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 19:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
0 siblings, 1 reply; 178+ messages in thread
From: Nathan Bossart @ 2026-03-11 19:02 UTC (permalink / raw)
To: Nazir Bilal Yavuz <byavuz81@gmail.com>; +Cc: Manni Wood <manni.wood@enterprisedb.com>; KAZAR Ayoub <ma_kazar@esi.dz>; Neil Conway <neil.conway@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
On Wed, Mar 11, 2026 at 09:49:22PM +0300, Nazir Bilal Yavuz wrote:
> That is what v13-0002 does. I saw 1%-2% speedups with this change and
> there was no regression.
Thanks for the explanation. Is there any reason _not_ to add this to 0001?
--
nathan
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 22:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-10 11:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-21 20:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Neil Conway <neil.conway@gmail.com>
2026-01-31 16:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 13:51 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 18:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 21:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-06 22:19 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 22:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-11 13:27 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-11 22:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-13 11:45 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-13 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 13:38 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-18 21:26 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 21:54 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 04:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-19 09:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 22:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-20 00:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-20 09:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-20 18:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-23 09:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 04:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-24 13:57 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 17:48 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-25 14:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-02 19:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-04 15:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-09 18:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-10 12:35 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-10 17:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 11:36 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 18:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 18:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 19:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
@ 2026-03-11 19:22 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 20:42 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
0 siblings, 1 reply; 178+ messages in thread
From: Nazir Bilal Yavuz @ 2026-03-11 19:22 UTC (permalink / raw)
To: Nathan Bossart <nathandbossart@gmail.com>; +Cc: Manni Wood <manni.wood@enterprisedb.com>; KAZAR Ayoub <ma_kazar@esi.dz>; Neil Conway <neil.conway@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
Hi,
On Wed, 11 Mar 2026 at 22:02, Nathan Bossart <nathandbossart@gmail.com> wrote:
>
> On Wed, Mar 11, 2026 at 09:49:22PM +0300, Nazir Bilal Yavuz wrote:
> > That is what v13-0002 does. I saw 1%-2% speedups with this change and
> > there was no regression.
>
> Thanks for the explanation. Is there any reason _not_ to add this to 0001?
I noticed this improvement today. To make it easier to review for
anyone who may have already started looking at v12, I attached the new
functional code changes as 0002. There was no other reason.
Here is v14 which is v13-0001 + v13-0002.
--
Regards,
Nazir Bilal Yavuz
Microsoft
Attachments:
[text/x-patch] v14-0001-Speed-up-COPY-FROM-text-CSV-parsing-using-SIMD.patch (9.8K, ../../CAN55FZ0yfETy4UEA5rOJ9S06JSOtiWF8TW_+yi3yjVAcrLqKLA@mail.gmail.com/2-v14-0001-Speed-up-COPY-FROM-text-CSV-parsing-using-SIMD.patch)
download | inline diff:
From d19e62275db2943cc4275cac2262c63d0bd4436c Mon Sep 17 00:00:00 2001
From: Nazir Bilal Yavuz <byavuz81@gmail.com>
Date: Wed, 11 Mar 2026 14:04:29 +0300
Subject: [PATCH v14] Speed up COPY FROM text/CSV parsing using SIMD
COPY FROM text and CSV parsing previously scanned the input buffer
one byte at a time looking for special characters (newline, carriage
return, backslash, and in CSV mode, quote and escape characters).
This patch adds a SIMD-accelerated fast path that processes a full
vector-width chunk per iteration, significantly reducing the number
of iterations needed on inputs with long lines.
A new helper function, CopyReadLineTextSIMDHelper(), loads chunks
of the input buffer into vector registers and checks for any special
characters using SIMD comparisons. When no special characters are
found, the entire chunk is skipped at once. When a special character
is found, the helper advances to that position and hands off to the
existing scalar code to handle it correctly.
To avoid a regression on inputs that contain many special characters,
SIMD is disabled for the remainder of the current input once a
non-EOL special character is encountered. SIMD is also disabled when
the processed line is shorter than a full vector, since SIMD can't
provide a benefit there.
Author: Shinya Kato <shinya11.kato@gmail.com>
Author: Nazir Bilal Yavuz <byavuz81@gmail.com>
Reviewed-by: Kazar Ayoub <ma_kazar@esi.dz>
Reviewed-by: Nathan Bossart <nathandbossart@gmail.com>
Reviewed-by: Neil Conway <neil.conway@gmail.com>
Reviewed-by: Andrew Dunstan <andrew@dunslane.net>
Reviewed-by: Manni Wood <manni.wood@enterprisedb.com>
Reviewed-by: Mark Wong <markwkm@gmail.com>
Discussion: https://postgr.es/m/CAOzEurSW8cNr6TPKsjrstnPfhf4QyQqB4tnPXGGe8N4e_v7Jig%40mail.gmail.com
---
src/backend/commands/copyfrom.c | 3 +
src/backend/commands/copyfromparse.c | 195 ++++++++++++++++++++++-
src/include/commands/copyfrom_internal.h | 3 +
3 files changed, 194 insertions(+), 7 deletions(-)
diff --git a/src/backend/commands/copyfrom.c b/src/backend/commands/copyfrom.c
index 2f42f55e229..fe18bd70890 100644
--- a/src/backend/commands/copyfrom.c
+++ b/src/backend/commands/copyfrom.c
@@ -1747,6 +1747,9 @@ BeginCopyFrom(ParseState *pstate,
cstate->cur_attval = NULL;
cstate->relname_only = false;
+ /* Initialize SIMD */
+ cstate->simd_enabled = true;
+
/*
* Allocate buffers for the input pipeline.
*
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index 84c8809a889..55159b0122c 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -72,6 +72,7 @@
#include "miscadmin.h"
#include "pgstat.h"
#include "port/pg_bswap.h"
+#include "port/simd.h"
#include "utils/builtins.h"
#include "utils/rel.h"
#include "utils/wait_event.h"
@@ -159,6 +160,12 @@ static pg_attribute_always_inline bool NextCopyFromRawFieldsInternal(CopyFromSta
int *nfields,
bool is_csv);
+/* SIMD functions */
+#ifndef USE_NO_SIMD
+static bool CopyReadLineTextSIMDHelper(CopyFromState cstate, bool is_csv,
+ bool *temp_hit_eof, int *temp_input_buf_ptr);
+#endif
+
/* Low-level communications functions */
static int CopyGetData(CopyFromState cstate, void *databuf,
@@ -1311,6 +1318,144 @@ CopyReadLine(CopyFromState cstate, bool is_csv)
return result;
}
+#ifndef USE_NO_SIMD
+/*
+ * Use SIMD instructions to efficiently scan the input buffer for special
+ * characters (e.g., newline, carriage return, quote, and escape). This is
+ * faster than byte-by-byte iteration, especially on large buffers.
+ *
+ * Note that, SIMD may become slower when the input contains many special
+ * characters. To avoid this regression, we disable SIMD for the rest of the
+ * input once we encounter a special character which isn't EOL.
+ * Also, SIMD is disabled when it encounters a short line that SIMD can't
+ * create a full sized vector, too.
+ */
+static bool
+CopyReadLineTextSIMDHelper(CopyFromState cstate, bool is_csv, bool *temp_hit_eof, int *temp_input_buf_ptr)
+{
+ char *copy_input_buf;
+ int input_buf_ptr;
+ int copy_buf_len;
+ bool result = false;
+ bool unique_escapec = false;
+ bool first_vector = true;
+ Vector8 nl = vector8_broadcast('\n');
+ Vector8 cr = vector8_broadcast('\r');
+ Vector8 bs_or_quote = vector8_broadcast('\\');
+ Vector8 escape = vector8_broadcast(0);
+
+ if (is_csv)
+ {
+ char quotec = cstate->opts.quote[0];
+ char escapec = cstate->opts.escape[0];
+
+ bs_or_quote = vector8_broadcast(quotec);
+ if (quotec != escapec)
+ {
+ unique_escapec = true;
+ escape = vector8_broadcast(escapec);
+ }
+ }
+
+ /* For a little extra speed we copy these into local variables */
+ copy_input_buf = cstate->input_buf;
+ input_buf_ptr = cstate->input_buf_index;
+ copy_buf_len = cstate->input_buf_len;
+
+ while (true)
+ {
+ /* Load more data if needed */
+ if (sizeof(Vector8) > copy_buf_len - input_buf_ptr)
+ {
+ REFILL_LINEBUF;
+
+ CopyLoadInputBuf(cstate);
+ /* update our local variables */
+ *temp_hit_eof = cstate->input_reached_eof;
+ input_buf_ptr = cstate->input_buf_index;
+ copy_buf_len = cstate->input_buf_len;
+
+ /*
+ * If we are completely out of data, break out of the loop,
+ * reporting EOF.
+ */
+ if (INPUT_BUF_BYTES(cstate) <= 0)
+ {
+ result = true;
+ break;
+ }
+ }
+
+ if (copy_buf_len - input_buf_ptr >= sizeof(Vector8))
+ {
+ Vector8 chunk;
+ Vector8 match = vector8_broadcast(0);
+
+ /* Load a chunk of data into a vector register */
+ vector8_load(&chunk, (const uint8 *) ©_input_buf[input_buf_ptr]);
+
+ match = vector8_or(vector8_eq(chunk, nl), vector8_eq(chunk, cr));
+ match = vector8_or(match, vector8_eq(chunk, bs_or_quote));
+ if (unique_escapec)
+ match = vector8_or(match, vector8_eq(chunk, escape));
+
+ /* Check if we found any special characters */
+ if (vector8_is_highbit_set(match))
+ {
+ /*
+ * Found a special character. Advance up to that point and let
+ * the scalar code handle it.
+ */
+ uint32 mask;
+ int advance;
+ char c;
+
+ mask = vector8_highbit_mask(match);
+ advance = pg_rightmost_one_pos32(mask);
+
+ input_buf_ptr += advance;
+ c = copy_input_buf[input_buf_ptr];
+
+ /*
+ * If we encountered a special character in the first vector,
+ * this means line is not long enough to skip fully sized
+ * vector. To be cautious, disable SIMD for the rest.
+ *
+ * Otherwise, do not disable SIMD when we hit EOL characters.
+ * We don't check for EOF because parsing ends there.
+ */
+ if (first_vector || !(c == '\r' || c == '\n'))
+ cstate->simd_enabled = false;
+
+ break;
+ }
+ else
+ {
+ /* No special characters found, so skip the entire chunk */
+ input_buf_ptr += sizeof(Vector8);
+ first_vector = false;
+ }
+ }
+
+ /*
+ * Although we refill linebuf, there is not enough character to fill
+ * full sized vector. This doesn't mean that we encountered a line
+ * that is not enough to fill a full sized vector.
+ *
+ * Scalar code will handle the rest for this line. Then, SIMD will
+ * continue from the next line.
+ */
+ else
+ {
+ break;
+ }
+ }
+
+ *temp_input_buf_ptr = input_buf_ptr;
+ return result;
+}
+#endif
+
/*
* CopyReadLineText - inner loop of CopyReadLine for text mode
*/
@@ -1339,6 +1484,49 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
escapec = '\0';
}
+ /*
+ * input_buf_ptr might be updated in the SIMD Helper function, so it needs
+ * to be set before calling CopyReadLineTextSIMDHelper().
+ */
+ input_buf_ptr = cstate->input_buf_index;
+
+#ifndef USE_NO_SIMD
+ /* First try to run SIMD, then continue with the scalar path */
+ if (cstate->simd_enabled)
+ {
+ /*
+ * Temporary variables are used here instead of passing the actual
+ * variables (especially input_buf_ptr) directly to the helper. Taking
+ * the address of a local variable might force the compiler to
+ * allocate it on the stack rather than in a register. Because
+ * input_buf_ptr is used heavily in the hot scalar path below, keeping
+ * it in a register is important for performance.
+ */
+ int temp_input_buf_ptr;
+ bool temp_hit_eof = hit_eof;
+
+ result = CopyReadLineTextSIMDHelper(cstate, is_csv, &temp_hit_eof,
+ &temp_input_buf_ptr);
+ input_buf_ptr = temp_input_buf_ptr;
+ hit_eof = temp_hit_eof;
+
+ /* Early exit from SIMD */
+ if (result)
+ {
+ /*
+ * Transfer any still-uncopied data to line_buf.
+ */
+ REFILL_LINEBUF;
+
+ return result;
+ }
+ }
+#endif
+
+ /* For a little extra speed we copy these into local variables */
+ copy_input_buf = cstate->input_buf;
+ copy_buf_len = cstate->input_buf_len;
+
/*
* The objective of this loop is to transfer the entire next input line
* into line_buf. Hence, we only care for detecting newlines (\r and/or
@@ -1360,14 +1548,7 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
* character to examine; any characters from input_buf_index to
* input_buf_ptr have been determined to be part of the line, but not yet
* transferred to line_buf.
- *
- * For a little extra speed within the loop, we copy input_buf and
- * input_buf_len into local variables.
*/
- copy_input_buf = cstate->input_buf;
- input_buf_ptr = cstate->input_buf_index;
- copy_buf_len = cstate->input_buf_len;
-
for (;;)
{
int prev_raw_ptr;
diff --git a/src/include/commands/copyfrom_internal.h b/src/include/commands/copyfrom_internal.h
index f892c343157..5b020bf4d0b 100644
--- a/src/include/commands/copyfrom_internal.h
+++ b/src/include/commands/copyfrom_internal.h
@@ -89,6 +89,9 @@ typedef struct CopyFromStateData
const char *cur_attval; /* current att value for error messages */
bool relname_only; /* don't output line number, att, etc. */
+ /* SIMD variables */
+ bool simd_enabled;
+
/*
* Working state
*/
--
2.47.3
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 22:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-10 11:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-21 20:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Neil Conway <neil.conway@gmail.com>
2026-01-31 16:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 13:51 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 18:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 21:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-06 22:19 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 22:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-11 13:27 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-11 22:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-13 11:45 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-13 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 13:38 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-18 21:26 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 21:54 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 04:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-19 09:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 22:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-20 00:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-20 09:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-20 18:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-23 09:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 04:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-24 13:57 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 17:48 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-25 14:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-02 19:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-04 15:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-09 18:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-10 12:35 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-10 17:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 11:36 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 18:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 18:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 19:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 19:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
@ 2026-03-11 20:42 ` Nathan Bossart <nathandbossart@gmail.com>
2026-03-12 10:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
0 siblings, 1 reply; 178+ messages in thread
From: Nathan Bossart @ 2026-03-11 20:42 UTC (permalink / raw)
To: Nazir Bilal Yavuz <byavuz81@gmail.com>; +Cc: Manni Wood <manni.wood@enterprisedb.com>; KAZAR Ayoub <ma_kazar@esi.dz>; Neil Conway <neil.conway@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
On Wed, Mar 11, 2026 at 10:22:18PM +0300, Nazir Bilal Yavuz wrote:
> Here is v14 which is v13-0001 + v13-0002.
Thanks! It's getting close.
> + /*
> + * Temporary variables are used here instead of passing the actual
> + * variables (especially input_buf_ptr) directly to the helper. Taking
> + * the address of a local variable might force the compiler to
> + * allocate it on the stack rather than in a register. Because
> + * input_buf_ptr is used heavily in the hot scalar path below, keeping
> + * it in a register is important for performance.
> + */
> + int temp_input_buf_ptr;
> + bool temp_hit_eof = hit_eof;
A few notes:
* Does using a temporary variable for hit_eof actually make a difference?
AFAICT that's only updated when loading more data.
* Does inlining the function produce the same results?
* Also, I'm curious what the usual benchmarks look like with and without
this hack for the latest patch.
--
nathan
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 22:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-10 11:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-21 20:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Neil Conway <neil.conway@gmail.com>
2026-01-31 16:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 13:51 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 18:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 21:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-06 22:19 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 22:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-11 13:27 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-11 22:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-13 11:45 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-13 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 13:38 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-18 21:26 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 21:54 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 04:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-19 09:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 22:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-20 00:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-20 09:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-20 18:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-23 09:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 04:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-24 13:57 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 17:48 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-25 14:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-02 19:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-04 15:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-09 18:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-10 12:35 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-10 17:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 11:36 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 18:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 18:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 19:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 19:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 20:42 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
@ 2026-03-12 10:59 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-12 17:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
0 siblings, 1 reply; 178+ messages in thread
From: Nazir Bilal Yavuz @ 2026-03-12 10:59 UTC (permalink / raw)
To: Nathan Bossart <nathandbossart@gmail.com>; +Cc: Manni Wood <manni.wood@enterprisedb.com>; KAZAR Ayoub <ma_kazar@esi.dz>; Neil Conway <neil.conway@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
Hi,
On Wed, 11 Mar 2026 at 23:42, Nathan Bossart <nathandbossart@gmail.com> wrote:
>
> On Wed, Mar 11, 2026 at 10:22:18PM +0300, Nazir Bilal Yavuz wrote:
> > Here is v14 which is v13-0001 + v13-0002.
>
> Thanks! It's getting close.
>
> > + /*
> > + * Temporary variables are used here instead of passing the actual
> > + * variables (especially input_buf_ptr) directly to the helper. Taking
> > + * the address of a local variable might force the compiler to
> > + * allocate it on the stack rather than in a register. Because
> > + * input_buf_ptr is used heavily in the hot scalar path below, keeping
> > + * it in a register is important for performance.
> > + */
> > + int temp_input_buf_ptr;
> > + bool temp_hit_eof = hit_eof;
>
> A few notes:
>
> * Does using a temporary variable for hit_eof actually make a difference?
> AFAICT that's only updated when loading more data.
>
> * Does inlining the function produce the same results?
>
> * Also, I'm curious what the usual benchmarks look like with and without
> this hack for the latest patch.
I tried to benchmark all of these questions, here are the results:
Old master means d841ca2d14 - inlining CopyReadLineText commit (dc592a4155).
v14 means d841ca2d14 + v14.
v14 + #1 means removing temporary variables.
v14 + #2 means removing temp_hit_eof variable only.
v14 + #3 means inlining CopyReadLineTextSIMDHelper().
v14 + #4 means inlining CopyReadLineTextSIMDHelper() + removing
temporary variables (#1).
------------------------------------------------------------
Results for default_toast_compression = 'lz4':
+-------------------------------------------+
| Optimization: -O2 |
+------------+--------------+---------------+
| | Text | CSV |
+------------+------+-------+-------+-------+
| WIDE | None | 1/3 | None | 1/3 |
+------------+------+-------+-------+-------+
| Old master | 4260 | 4789 | 5930 | 8276 |
+------------+------+-------+-------+-------+
| v14 | 2489 | 4439 | 2529 | 8098 |
+------------+------+-------+-------+-------+
| v14 + #1 | 2472 | 5177 | 2479 | 9285 |
+------------+------+-------+-------+-------+
| v14 + #2 | 2521 | 4252 | 2481 | 8050 |
+------------+------+-------+-------+-------+
| v14 + #3 | 2632 | 4569 | 2458 | 8657 |
+------------+------+-------+-------+-------+
| v14 + #4 | 2476 | 4239 | 2475 | 10544 |
+------------+------+-------+-------+-------+
| | | | | |
+------------+------+-------+-------+-------+
| | | | | |
+------------+------+-------+-------+-------+
| | Text | CSV |
+------------+------+-------+-------+-------+
| NARROW | None | 1/3 | None | 1/3 |
+------------+------+-------+-------+-------+
| Old master | 9955 | 10056 | 10329 | 10872 |
+------------+------+-------+-------+-------+
| v14 | 9917 | 10080 | 10104 | 10510 |
+------------+------+-------+-------+-------+
| v14 + #1 | 9913 | 10090 | 10120 | 10532 |
+------------+------+-------+-------+-------+
| v14 + #2 | 9937 | 10130 | 10072 | 10520 |
+------------+------+-------+-------+-------+
| v14 + #3 | 9880 | 10258 | 10220 | 10604 |
+------------+------+-------+-------+-------+
| v14 + #4 | 9827 | 10306 | 10308 | 10734 |
+------------+------+-------+-------+-------+
------------------------------------------------------------
Results for default_toast_compression = 'pglz':
+-------------------------------------------+
| Optimization: -O2 |
+------------+--------------+---------------+
| | Text | CSV |
+------------+------+-------+-------+-------+
| WIDE | None | 1/3 | None | 1/3 |
+------------+------+-------+-------+-------+
| Old master | 4260 | 4789 | 5930 | 8276 |
+------------+------+-------+-------+-------+
| v14 | 2489 | 4439 | 2529 | 8098 |
+------------+------+-------+-------+-------+
| v14 + #1 | 2472 | 5177 | 2479 | 9285 |
+------------+------+-------+-------+-------+
| v14 + #2 | 2521 | 4252 | 2481 | 8050 |
+------------+------+-------+-------+-------+
| v14 + #3 | 2632 | 4569 | 2458 | 8657 |
+------------+------+-------+-------+-------+
| v14 + #4 | 2476 | 4239 | 2475 | 10544 |
+------------+------+-------+-------+-------+
| | | | | |
+------------+------+-------+-------+-------+
| | | | | |
+------------+------+-------+-------+-------+
| | Text | CSV |
+------------+------+-------+-------+-------+
| NARROW | None | 1/3 | None | 1/3 |
+------------+------+-------+-------+-------+
| Old master | 9955 | 10056 | 10329 | 10872 |
+------------+------+-------+-------+-------+
| v14 | 9917 | 10080 | 10104 | 10510 |
+------------+------+-------+-------+-------+
| v14 + #1 | 9913 | 10090 | 10120 | 10532 |
+------------+------+-------+-------+-------+
| v14 + #2 | 9937 | 10130 | 10072 | 10520 |
+------------+------+-------+-------+-------+
| v14 + #3 | 9880 | 10258 | 10220 | 10604 |
+------------+------+-------+-------+-------+
| v14 + #4 | 9827 | 10306 | 10308 | 10734 |
+------------+------+-------+-------+-------+
------------------------------------------------------------
By looking these results:
v14 + #1 and v14 + #3 performs worse on wide & 1/3 cases.
v14 + #4 performs worse on CSV & wide & 1/3 cases.
v14 and v14 + #2 perform very similarly. They don't have regression. I
think we can move forward with one of these.
--
Regards,
Nazir Bilal Yavuz
Microsoft
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 22:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-10 11:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-21 20:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Neil Conway <neil.conway@gmail.com>
2026-01-31 16:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 13:51 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 18:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 21:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-06 22:19 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 22:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-11 13:27 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-11 22:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-13 11:45 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-13 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 13:38 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-18 21:26 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 21:54 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 04:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-19 09:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 22:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-20 00:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-20 09:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-20 18:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-23 09:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 04:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-24 13:57 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 17:48 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-25 14:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-02 19:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-04 15:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-09 18:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-10 12:35 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-10 17:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 11:36 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 18:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 18:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 19:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 19:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 20:42 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-12 10:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
@ 2026-03-12 17:37 ` Nathan Bossart <nathandbossart@gmail.com>
2026-03-13 02:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-03-13 11:57 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
0 siblings, 2 replies; 178+ messages in thread
From: Nathan Bossart @ 2026-03-12 17:37 UTC (permalink / raw)
To: Nazir Bilal Yavuz <byavuz81@gmail.com>; +Cc: Manni Wood <manni.wood@enterprisedb.com>; KAZAR Ayoub <ma_kazar@esi.dz>; Neil Conway <neil.conway@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
Here is what I have staged for commit, which I'm planning to do tomorrow.
Please review and/or test if you are able.
--
nathan
From 023f4a27f69bc5ed30d6678d520aace84642ca0d Mon Sep 17 00:00:00 2001
From: Nathan Bossart <nathan@postgresql.org>
Date: Thu, 12 Mar 2026 12:32:23 -0500
Subject: [PATCH v15 1/1] Optimize COPY FROM (FORMAT {text,csv}) using SIMD.
Presently, such commands scan the input buffer one byte at a time
looking for special characters. This commit adds a new path that
uses SIMD instructions to skip over chunks of data without any
special characters. This can be much faster.
To avoid regressions, SIMD processing is disabled for the remainder
of the COPY FROM command as soon as we encounter a short line or a
special character (except for end-of-line characters, else we'd
always disable it after the first line). This is perhaps too
conservative, but it could probably be made more lenient in the
future via fine-tuned heuristics.
Author: Nazir Bilal Yavuz <byavuz81@gmail.com>
Co-authored-by: Shinya Kato <shinya11.kato@gmail.com>
Reviewed-by: Ayoub Kazar <ma_kazar@esi.dz>
Reviewed-by: Andrew Dunstan <andrew@dunslane.net>
Reviewed-by: Neil Conway <neil.conway@gmail.com>
Tested-by: Manni Wood <manni.wood@enterprisedb.com>
Tested-by: Mark Wong <markwkm@gmail.com>
Discussion: https://postgr.es/m/CAOzEurSW8cNr6TPKsjrstnPfhf4QyQqB4tnPXGGe8N4e_v7Jig%40mail.gmail.com
---
src/backend/commands/copyfrom.c | 1 +
src/backend/commands/copyfromparse.c | 180 ++++++++++++++++++++++-
src/include/commands/copyfrom_internal.h | 1 +
3 files changed, 179 insertions(+), 3 deletions(-)
diff --git a/src/backend/commands/copyfrom.c b/src/backend/commands/copyfrom.c
index 2f42f55e229..4d0a04db848 100644
--- a/src/backend/commands/copyfrom.c
+++ b/src/backend/commands/copyfrom.c
@@ -1746,6 +1746,7 @@ BeginCopyFrom(ParseState *pstate,
cstate->cur_attname = NULL;
cstate->cur_attval = NULL;
cstate->relname_only = false;
+ cstate->simd_enabled = true;
/*
* Allocate buffers for the input pipeline.
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index 84c8809a889..5ee08168884 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -72,6 +72,7 @@
#include "miscadmin.h"
#include "pgstat.h"
#include "port/pg_bswap.h"
+#include "port/simd.h"
#include "utils/builtins.h"
#include "utils/rel.h"
#include "utils/wait_event.h"
@@ -1311,6 +1312,152 @@ CopyReadLine(CopyFromState cstate, bool is_csv)
return result;
}
+#ifndef USE_NO_SIMD
+/*
+ * Helper function for CopyReadLineText() that uses SIMD instructions to scan
+ * the input buffer for special characters. This can be much faster.
+ *
+ * Note that we disable SIMD for the remainder of the COPY FROM command upon
+ * encountering a special character (except for end-of-line characters) or a
+ * short line. This is perhaps too conservative, but it should help avoid
+ * regressions. It could probably be made more lenient in the future via
+ * fine-tuned heuristics.
+ */
+static bool
+CopyReadLineTextSIMDHelper(CopyFromState cstate, bool is_csv,
+ bool *hit_eof_p, int *input_buf_ptr_p)
+{
+ char *copy_input_buf;
+ int input_buf_ptr;
+ int copy_buf_len;
+ bool unique_esc_char; /* for csv, do quote/esc chars differ? */
+ bool first = true;
+ bool result = false;
+ const Vector8 nl_vec = vector8_broadcast('\n');
+ const Vector8 cr_vec = vector8_broadcast('\r');
+ Vector8 bs_or_quote_vec; /* '\' for text, quote for csv */
+ Vector8 esc_vec; /* only for csv */
+
+ if (is_csv)
+ {
+ char quote = cstate->opts.quote[0];
+ char esc = cstate->opts.escape[0];
+
+ bs_or_quote_vec = vector8_broadcast(quote);
+ esc_vec = vector8_broadcast(esc);
+ unique_esc_char = (quote != esc);
+ }
+ else
+ {
+ bs_or_quote_vec = vector8_broadcast('\\');
+ unique_esc_char = false;
+ }
+
+ /*
+ * For a little extra speed within the loop, we copy some state members
+ * into local variables. Note that we need to use a separate local
+ * variable for input_buf_ptr so that the REFILL_LINEBUF macro works. We
+ * copy its value into the input_buf_ptr_p argument before returning.
+ */
+ copy_input_buf = cstate->input_buf;
+ input_buf_ptr = cstate->input_buf_index;
+ copy_buf_len = cstate->input_buf_len;
+
+ /*
+ * See the corresponding loop in CopyReadLineText() for more information
+ * about the purpose of this loop. This one does the same thing using
+ * SIMD instructions, although we are quick to bail out to the scalar path
+ * if we encounter a special character.
+ */
+ for (;;)
+ {
+ Vector8 chunk;
+ Vector8 match;
+
+ /* Load more data if needed. */
+ if (copy_buf_len - input_buf_ptr < sizeof(Vector8))
+ {
+ REFILL_LINEBUF;
+
+ CopyLoadInputBuf(cstate);
+ /* update our local variables */
+ *hit_eof_p = cstate->input_reached_eof;
+ input_buf_ptr = cstate->input_buf_index;
+ copy_buf_len = cstate->input_buf_len;
+
+ /*
+ * If we are completely out of data, break out of the loop,
+ * reporting EOF.
+ */
+ if (INPUT_BUF_BYTES(cstate) <= 0)
+ {
+ result = true;
+ break;
+ }
+ }
+
+ /*
+ * If we still don't have enough data for the SIMD path, fall back to
+ * the scalar code. Note that this doesn't necessarily mean we
+ * encountered a short line, so we leave cstate->simd_enabled set to
+ * true.
+ */
+ if (copy_buf_len - input_buf_ptr < sizeof(Vector8))
+ break;
+
+ /*
+ * If we made it here, we have at least enough data to fit in a
+ * Vector8, so we can use SIMD instructions to scan for special
+ * characters.
+ */
+ vector8_load(&chunk, (const uint8 *) ©_input_buf[input_buf_ptr]);
+
+ /*
+ * Check for \n, \r, \\ (for text), quotes (for csv), and escapes (for
+ * csv, if different from quotes).
+ */
+ match = vector8_eq(chunk, nl_vec);
+ match = vector8_or(match, vector8_eq(chunk, cr_vec));
+ match = vector8_or(match, vector8_eq(chunk, bs_or_quote_vec));
+ if (unique_esc_char)
+ match = vector8_or(match, vector8_eq(chunk, esc_vec));
+
+ /*
+ * If we found a special character, advance to it and hand off to the
+ * scalar path. Except for end-of-line characters, we also disable
+ * SIMD processing for the remainder of the COPY FROM command.
+ */
+ if (vector8_is_highbit_set(match))
+ {
+ uint32 mask;
+ char c;
+
+ mask = vector8_highbit_mask(match);
+ input_buf_ptr += pg_rightmost_one_pos32(mask);
+
+ /*
+ * Don't disable SIMD if we found \n or \r, else we'd stop using
+ * SIMD instructions after the first line. As an exception, we do
+ * disable it if this is the first vector we processed, as that
+ * means the line is too short for SIMD.
+ */
+ c = copy_input_buf[input_buf_ptr];
+ if (first || (c != '\n' && c != '\r'))
+ cstate->simd_enabled = false;
+
+ break;
+ }
+
+ /* That chunk was clear of special characters, so we can skip it. */
+ input_buf_ptr += sizeof(Vector8);
+ first = false;
+ }
+
+ *input_buf_ptr_p = input_buf_ptr;
+ return result;
+}
+#endif /* ! USE_NO_SIMD */
+
/*
* CopyReadLineText - inner loop of CopyReadLine for text mode
*/
@@ -1361,11 +1508,36 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
* input_buf_ptr have been determined to be part of the line, but not yet
* transferred to line_buf.
*
- * For a little extra speed within the loop, we copy input_buf and
- * input_buf_len into local variables.
+ * For a little extra speed within the loop, we copy some state
+ * information into local variables. input_buf_ptr could be changed in
+ * the SIMD path, so we must set that one before it. The others are set
+ * afterwards.
*/
- copy_input_buf = cstate->input_buf;
input_buf_ptr = cstate->input_buf_index;
+
+ /*
+ * We first try to use SIMD for the task described above, falling back to
+ * the scalar path (i.e., the loop below) if needed.
+ */
+#ifndef USE_NO_SIMD
+ if (cstate->simd_enabled)
+ {
+ /*
+ * Using a temporary variable seems to encourage the compiler to keep
+ * it in a register, which is beneficial for performance.
+ */
+ int tmp_input_buf_ptr;
+
+ result = CopyReadLineTextSIMDHelper(cstate, is_csv, &hit_eof,
+ &tmp_input_buf_ptr);
+ input_buf_ptr = tmp_input_buf_ptr;
+
+ if (result)
+ goto out;
+ }
+#endif /* ! USE_NO_SIMD */
+
+ copy_input_buf = cstate->input_buf;
copy_buf_len = cstate->input_buf_len;
for (;;)
@@ -1605,6 +1777,8 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
}
} /* end of outer loop */
+out:
+
/*
* Transfer any still-uncopied data to line_buf.
*/
diff --git a/src/include/commands/copyfrom_internal.h b/src/include/commands/copyfrom_internal.h
index f892c343157..9d3e244ee55 100644
--- a/src/include/commands/copyfrom_internal.h
+++ b/src/include/commands/copyfrom_internal.h
@@ -108,6 +108,7 @@ typedef struct CopyFromStateData
* att */
bool *defaults; /* if DEFAULT marker was found for
* corresponding att */
+ bool simd_enabled; /* use SIMD to scan for special chars? */
/*
* True if the corresponding attribute's is a constrained domain. This
--
2.50.1 (Apple Git-155)
Attachments:
[text/plain] v15-0001-Optimize-COPY-FROM-FORMAT-text-csv-using-SIMD.patch (8.9K, ../../abL5ylYjmopRRmvy@nathan/2-v15-0001-Optimize-COPY-FROM-FORMAT-text-csv-using-SIMD.patch)
download | inline diff:
From 023f4a27f69bc5ed30d6678d520aace84642ca0d Mon Sep 17 00:00:00 2001
From: Nathan Bossart <nathan@postgresql.org>
Date: Thu, 12 Mar 2026 12:32:23 -0500
Subject: [PATCH v15 1/1] Optimize COPY FROM (FORMAT {text,csv}) using SIMD.
Presently, such commands scan the input buffer one byte at a time
looking for special characters. This commit adds a new path that
uses SIMD instructions to skip over chunks of data without any
special characters. This can be much faster.
To avoid regressions, SIMD processing is disabled for the remainder
of the COPY FROM command as soon as we encounter a short line or a
special character (except for end-of-line characters, else we'd
always disable it after the first line). This is perhaps too
conservative, but it could probably be made more lenient in the
future via fine-tuned heuristics.
Author: Nazir Bilal Yavuz <byavuz81@gmail.com>
Co-authored-by: Shinya Kato <shinya11.kato@gmail.com>
Reviewed-by: Ayoub Kazar <ma_kazar@esi.dz>
Reviewed-by: Andrew Dunstan <andrew@dunslane.net>
Reviewed-by: Neil Conway <neil.conway@gmail.com>
Tested-by: Manni Wood <manni.wood@enterprisedb.com>
Tested-by: Mark Wong <markwkm@gmail.com>
Discussion: https://postgr.es/m/CAOzEurSW8cNr6TPKsjrstnPfhf4QyQqB4tnPXGGe8N4e_v7Jig%40mail.gmail.com
---
src/backend/commands/copyfrom.c | 1 +
src/backend/commands/copyfromparse.c | 180 ++++++++++++++++++++++-
src/include/commands/copyfrom_internal.h | 1 +
3 files changed, 179 insertions(+), 3 deletions(-)
diff --git a/src/backend/commands/copyfrom.c b/src/backend/commands/copyfrom.c
index 2f42f55e229..4d0a04db848 100644
--- a/src/backend/commands/copyfrom.c
+++ b/src/backend/commands/copyfrom.c
@@ -1746,6 +1746,7 @@ BeginCopyFrom(ParseState *pstate,
cstate->cur_attname = NULL;
cstate->cur_attval = NULL;
cstate->relname_only = false;
+ cstate->simd_enabled = true;
/*
* Allocate buffers for the input pipeline.
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index 84c8809a889..5ee08168884 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -72,6 +72,7 @@
#include "miscadmin.h"
#include "pgstat.h"
#include "port/pg_bswap.h"
+#include "port/simd.h"
#include "utils/builtins.h"
#include "utils/rel.h"
#include "utils/wait_event.h"
@@ -1311,6 +1312,152 @@ CopyReadLine(CopyFromState cstate, bool is_csv)
return result;
}
+#ifndef USE_NO_SIMD
+/*
+ * Helper function for CopyReadLineText() that uses SIMD instructions to scan
+ * the input buffer for special characters. This can be much faster.
+ *
+ * Note that we disable SIMD for the remainder of the COPY FROM command upon
+ * encountering a special character (except for end-of-line characters) or a
+ * short line. This is perhaps too conservative, but it should help avoid
+ * regressions. It could probably be made more lenient in the future via
+ * fine-tuned heuristics.
+ */
+static bool
+CopyReadLineTextSIMDHelper(CopyFromState cstate, bool is_csv,
+ bool *hit_eof_p, int *input_buf_ptr_p)
+{
+ char *copy_input_buf;
+ int input_buf_ptr;
+ int copy_buf_len;
+ bool unique_esc_char; /* for csv, do quote/esc chars differ? */
+ bool first = true;
+ bool result = false;
+ const Vector8 nl_vec = vector8_broadcast('\n');
+ const Vector8 cr_vec = vector8_broadcast('\r');
+ Vector8 bs_or_quote_vec; /* '\' for text, quote for csv */
+ Vector8 esc_vec; /* only for csv */
+
+ if (is_csv)
+ {
+ char quote = cstate->opts.quote[0];
+ char esc = cstate->opts.escape[0];
+
+ bs_or_quote_vec = vector8_broadcast(quote);
+ esc_vec = vector8_broadcast(esc);
+ unique_esc_char = (quote != esc);
+ }
+ else
+ {
+ bs_or_quote_vec = vector8_broadcast('\\');
+ unique_esc_char = false;
+ }
+
+ /*
+ * For a little extra speed within the loop, we copy some state members
+ * into local variables. Note that we need to use a separate local
+ * variable for input_buf_ptr so that the REFILL_LINEBUF macro works. We
+ * copy its value into the input_buf_ptr_p argument before returning.
+ */
+ copy_input_buf = cstate->input_buf;
+ input_buf_ptr = cstate->input_buf_index;
+ copy_buf_len = cstate->input_buf_len;
+
+ /*
+ * See the corresponding loop in CopyReadLineText() for more information
+ * about the purpose of this loop. This one does the same thing using
+ * SIMD instructions, although we are quick to bail out to the scalar path
+ * if we encounter a special character.
+ */
+ for (;;)
+ {
+ Vector8 chunk;
+ Vector8 match;
+
+ /* Load more data if needed. */
+ if (copy_buf_len - input_buf_ptr < sizeof(Vector8))
+ {
+ REFILL_LINEBUF;
+
+ CopyLoadInputBuf(cstate);
+ /* update our local variables */
+ *hit_eof_p = cstate->input_reached_eof;
+ input_buf_ptr = cstate->input_buf_index;
+ copy_buf_len = cstate->input_buf_len;
+
+ /*
+ * If we are completely out of data, break out of the loop,
+ * reporting EOF.
+ */
+ if (INPUT_BUF_BYTES(cstate) <= 0)
+ {
+ result = true;
+ break;
+ }
+ }
+
+ /*
+ * If we still don't have enough data for the SIMD path, fall back to
+ * the scalar code. Note that this doesn't necessarily mean we
+ * encountered a short line, so we leave cstate->simd_enabled set to
+ * true.
+ */
+ if (copy_buf_len - input_buf_ptr < sizeof(Vector8))
+ break;
+
+ /*
+ * If we made it here, we have at least enough data to fit in a
+ * Vector8, so we can use SIMD instructions to scan for special
+ * characters.
+ */
+ vector8_load(&chunk, (const uint8 *) ©_input_buf[input_buf_ptr]);
+
+ /*
+ * Check for \n, \r, \\ (for text), quotes (for csv), and escapes (for
+ * csv, if different from quotes).
+ */
+ match = vector8_eq(chunk, nl_vec);
+ match = vector8_or(match, vector8_eq(chunk, cr_vec));
+ match = vector8_or(match, vector8_eq(chunk, bs_or_quote_vec));
+ if (unique_esc_char)
+ match = vector8_or(match, vector8_eq(chunk, esc_vec));
+
+ /*
+ * If we found a special character, advance to it and hand off to the
+ * scalar path. Except for end-of-line characters, we also disable
+ * SIMD processing for the remainder of the COPY FROM command.
+ */
+ if (vector8_is_highbit_set(match))
+ {
+ uint32 mask;
+ char c;
+
+ mask = vector8_highbit_mask(match);
+ input_buf_ptr += pg_rightmost_one_pos32(mask);
+
+ /*
+ * Don't disable SIMD if we found \n or \r, else we'd stop using
+ * SIMD instructions after the first line. As an exception, we do
+ * disable it if this is the first vector we processed, as that
+ * means the line is too short for SIMD.
+ */
+ c = copy_input_buf[input_buf_ptr];
+ if (first || (c != '\n' && c != '\r'))
+ cstate->simd_enabled = false;
+
+ break;
+ }
+
+ /* That chunk was clear of special characters, so we can skip it. */
+ input_buf_ptr += sizeof(Vector8);
+ first = false;
+ }
+
+ *input_buf_ptr_p = input_buf_ptr;
+ return result;
+}
+#endif /* ! USE_NO_SIMD */
+
/*
* CopyReadLineText - inner loop of CopyReadLine for text mode
*/
@@ -1361,11 +1508,36 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
* input_buf_ptr have been determined to be part of the line, but not yet
* transferred to line_buf.
*
- * For a little extra speed within the loop, we copy input_buf and
- * input_buf_len into local variables.
+ * For a little extra speed within the loop, we copy some state
+ * information into local variables. input_buf_ptr could be changed in
+ * the SIMD path, so we must set that one before it. The others are set
+ * afterwards.
*/
- copy_input_buf = cstate->input_buf;
input_buf_ptr = cstate->input_buf_index;
+
+ /*
+ * We first try to use SIMD for the task described above, falling back to
+ * the scalar path (i.e., the loop below) if needed.
+ */
+#ifndef USE_NO_SIMD
+ if (cstate->simd_enabled)
+ {
+ /*
+ * Using a temporary variable seems to encourage the compiler to keep
+ * it in a register, which is beneficial for performance.
+ */
+ int tmp_input_buf_ptr;
+
+ result = CopyReadLineTextSIMDHelper(cstate, is_csv, &hit_eof,
+ &tmp_input_buf_ptr);
+ input_buf_ptr = tmp_input_buf_ptr;
+
+ if (result)
+ goto out;
+ }
+#endif /* ! USE_NO_SIMD */
+
+ copy_input_buf = cstate->input_buf;
copy_buf_len = cstate->input_buf_len;
for (;;)
@@ -1605,6 +1777,8 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
}
} /* end of outer loop */
+out:
+
/*
* Transfer any still-uncopied data to line_buf.
*/
diff --git a/src/include/commands/copyfrom_internal.h b/src/include/commands/copyfrom_internal.h
index f892c343157..9d3e244ee55 100644
--- a/src/include/commands/copyfrom_internal.h
+++ b/src/include/commands/copyfrom_internal.h
@@ -108,6 +108,7 @@ typedef struct CopyFromStateData
* att */
bool *defaults; /* if DEFAULT marker was found for
* corresponding att */
+ bool simd_enabled; /* use SIMD to scan for special chars? */
/*
* True if the corresponding attribute's is a constrained domain. This
--
2.50.1 (Apple Git-155)
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 22:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-10 11:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-21 20:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Neil Conway <neil.conway@gmail.com>
2026-01-31 16:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 13:51 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 18:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 21:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-06 22:19 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 22:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-11 13:27 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-11 22:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-13 11:45 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-13 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 13:38 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-18 21:26 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 21:54 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 04:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-19 09:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 22:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-20 00:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-20 09:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-20 18:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-23 09:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 04:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-24 13:57 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 17:48 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-25 14:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-02 19:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-04 15:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-09 18:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-10 12:35 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-10 17:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 11:36 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 18:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 18:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 19:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 19:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 20:42 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-12 10:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-12 17:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
@ 2026-03-13 02:39 ` Manni Wood <manni.wood@enterprisedb.com>
2026-03-13 15:00 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
1 sibling, 1 reply; 178+ messages in thread
From: Manni Wood @ 2026-03-13 02:39 UTC (permalink / raw)
To: Nathan Bossart <nathandbossart@gmail.com>; +Cc: Nazir Bilal Yavuz <byavuz81@gmail.com>; KAZAR Ayoub <ma_kazar@esi.dz>; Neil Conway <neil.conway@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
On Thu, Mar 12, 2026 at 12:37 PM Nathan Bossart <nathandbossart@gmail.com>
wrote:
> Here is what I have staged for commit, which I'm planning to do tomorrow.
> Please review and/or test if you are able.
>
> --
> nathan
>
Hello, Nathan!
I found some time this evening to run some benchmarks using your v15 patch.
I hope these help.
lz4 - arm
arm NARROW master (git revert dc592a41557b072178f1798700bf9c69cd8e4235)
default_toast_compression = lz4
TXT : 10203.493250 ms
CSV : 10217.946000 ms
TXT with 1/3 escapes: 10305.912750 ms
CSV with 1/3 quotes: 12339.182000 ms
arm NARROW v15 default_toast_compression = lz4
TXT : 10205.261500 ms -0.017330% regression
CSV : 10358.898500 ms -1.379460% regression
TXT with 1/3 escapes: 10053.073000 ms 2.453347% improvement
CSV with 1/3 quotes: 11881.337000 ms 3.710497% improvement
arm WIDE master (git revert dc592a41557b072178f1798700bf9c69cd8e4235)
default_toast_compression = lz4
TXT : 5613.525250 ms
CSV : 8069.692750 ms
TXT with 1/3 escapes: 7088.888250 ms
CSV with 1/3 quotes: 10902.545500 ms
arm WIDE v15 default_toast_compression = lz4
TXT : 3201.494500 ms 42.968200% improvement
CSV : 3146.033750 ms 61.014207% improvement
TXT with 1/3 escapes: 6677.907500 ms 5.797535% improvement
CSV with 1/3 quotes: 10766.909500 ms 1.244076% improvement
lz4 - x86
x86 NARROW master (git revert dc592a41557b072178f1798700bf9c69cd8e4235)
default_toast_compression = lz4
TXT : 26110.287750 ms
CSV : 27923.199750 ms
TXT with 1/3 escapes: 27984.483250 ms
CSV with 1/3 quotes: 34387.239000 ms
x86 NARROW v15 default_toast_compression = lz4
TXT : 26019.629000 ms 0.347215% improvement
CSV : 26379.889000 ms 5.526984% improvement
TXT with 1/3 escapes: 28865.322750 ms -3.147600% regression
CSV with 1/3 quotes: 33218.293250 ms 3.399359% improvement
x86 WIDE master (git revert dc592a41557b072178f1798700bf9c69cd8e4235)
default_toast_compression = lz4
TXT : 15829.765000 ms
CSV : 20479.146000 ms
TXT with 1/3 escapes: 18437.507500 ms
CSV with 1/3 quotes: 29749.379250 ms
x86 WIDE v15 default_toast_compression = lz4
TXT : 8056.305000 ms 49.106604% improvement
CSV : 7997.555500 ms 60.947808% improvement
TXT with 1/3 escapes: 16324.925500 ms 11.458067% improvement
CSV with 1/3 quotes: 29978.346500 ms -0.769654% regression
pglz - arm
arm NARROW master (git revert dc592a41557b072178f1798700bf9c69cd8e4235)
default_toast_compression = pglz
TXT : 10334.666250 ms
CSV : 10978.851250 ms
TXT with 1/3 escapes: 11076.502750 ms
CSV with 1/3 quotes: 12582.679000 ms
arm NARROW v15 default_toast_compression = pglz
TXT : 10002.507750 ms 3.214023% improvement
CSV : 10017.436250 ms 8.756973% improvement
TXT with 1/3 escapes: 10179.949000 ms 8.094195% improvement
CSV with 1/3 quotes: 12088.836750 ms 3.924778% improvement
arm WIDE master (git revert dc592a41557b072178f1798700bf9c69cd8e4235)
default_toast_compression = pglz
TXT : 11403.206000 ms
CSV : 13915.718750 ms
TXT with 1/3 escapes: 12888.060250 ms
CSV with 1/3 quotes: 16741.463000 ms
arm WIDE v15 default_toast_compression = pglz
TXT : 9005.868250 ms 21.023366% improvement
CSV : 8935.159250 ms 35.790889% improvement
TXT with 1/3 escapes: 12432.655250 ms 3.533542% improvement
CSV with 1/3 quotes: 16564.852250 ms 1.054930% improvement
pglz - x86
x86 NARROW master (git revert dc592a41557b072178f1798700bf9c69cd8e4235)
default_toast_compression = pglz
TXT : 26404.516250 ms
CSV : 28138.719000 ms
TXT with 1/3 escapes: 28084.379750 ms
CSV with 1/3 quotes: 34502.702250 ms
x86 NARROW v15 default_toast_compression = pglz
TXT : 26438.415000 ms -0.128382% regression
CSV : 26869.718000 ms 4.509804% improvement
TXT with 1/3 escapes: 29379.299750 ms -4.610819% regression
CSV with 1/3 quotes: 33371.390250 ms 3.278908% improvement
x86 WIDE master (git revert dc592a41557b072178f1798700bf9c69cd8e4235)
default_toast_compression = pglz
TXT : 30595.372000 ms
CSV : 35665.908500 ms
TXT with 1/3 escapes: 32746.252000 ms
CSV with 1/3 quotes: 44136.542750 ms
x86 WIDE v15 default_toast_compression = pglz
TXT : 22681.770750 ms 25.865354% improvement
CSV : 22692.153000 ms 36.375789% improvement
TXT with 1/3 escapes: 30638.978000 ms 6.435161% improvement
CSV with 1/3 quotes: 44330.233000 ms -0.438843% regression
--
-- Manni Wood EDB: https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 22:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-10 11:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-21 20:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Neil Conway <neil.conway@gmail.com>
2026-01-31 16:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 13:51 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 18:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 21:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-06 22:19 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 22:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-11 13:27 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-11 22:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-13 11:45 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-13 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 13:38 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-18 21:26 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 21:54 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 04:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-19 09:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 22:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-20 00:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-20 09:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-20 18:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-23 09:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 04:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-24 13:57 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 17:48 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-25 14:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-02 19:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-04 15:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-09 18:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-10 12:35 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-10 17:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 11:36 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 18:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 18:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 19:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 19:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 20:42 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-12 10:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-12 17:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-13 02:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
@ 2026-03-13 15:00 ` Nathan Bossart <nathandbossart@gmail.com>
0 siblings, 0 replies; 178+ messages in thread
From: Nathan Bossart @ 2026-03-13 15:00 UTC (permalink / raw)
To: Manni Wood <manni.wood@enterprisedb.com>; +Cc: Nazir Bilal Yavuz <byavuz81@gmail.com>; KAZAR Ayoub <ma_kazar@esi.dz>; Neil Conway <neil.conway@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
On Thu, Mar 12, 2026 at 09:39:38PM -0500, Manni Wood wrote:
> I found some time this evening to run some benchmarks using your v15 patch.
> I hope these help.
Thanks!
> x86 NARROW v15 default_toast_compression = lz4
> TXT : 26019.629000 ms 0.347215% improvement
> CSV : 26379.889000 ms 5.526984% improvement
> TXT with 1/3 escapes: 28865.322750 ms -3.147600% regression
> CSV with 1/3 quotes: 33218.293250 ms 3.399359% improvement
> x86 NARROW v15 default_toast_compression = pglz
> TXT : 26438.415000 ms -0.128382% regression
> CSV : 26869.718000 ms 4.509804% improvement
> TXT with 1/3 escapes: 29379.299750 ms -4.610819% regression
> CSV with 1/3 quotes: 33371.390250 ms 3.278908% improvement
Those 3-5% regressions are interesting, but given there are similar
"improvements" for the surrounding cases, I'm going to consider them as
noise for now and proceed with the patch. If folks feel strongly about
digging deeper here, I'm happy to revisit the subject.
--
nathan
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 22:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-10 11:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-21 20:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Neil Conway <neil.conway@gmail.com>
2026-01-31 16:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 13:51 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 18:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 21:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-06 22:19 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 22:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-11 13:27 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-11 22:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-13 11:45 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-13 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 13:38 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-18 21:26 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 21:54 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 04:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-19 09:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 22:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-20 00:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-20 09:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-20 18:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-23 09:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 04:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-24 13:57 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 17:48 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-25 14:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-02 19:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-04 15:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-09 18:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-10 12:35 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-10 17:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 11:36 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 18:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 18:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 19:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 19:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 20:42 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-12 10:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-12 17:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
@ 2026-03-13 11:57 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-13 13:34 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
1 sibling, 1 reply; 178+ messages in thread
From: Nazir Bilal Yavuz @ 2026-03-13 11:57 UTC (permalink / raw)
To: Nathan Bossart <nathandbossart@gmail.com>; +Cc: Manni Wood <manni.wood@enterprisedb.com>; KAZAR Ayoub <ma_kazar@esi.dz>; Neil Conway <neil.conway@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
Hi,
On Thu, 12 Mar 2026 at 20:37, Nathan Bossart <nathandbossart@gmail.com> wrote:
>
> Here is what I have staged for commit, which I'm planning to do tomorrow.
> Please review and/or test if you are able.
Thank you!
Unfortunately, v15 causes a regression for a 'csv & wide & 1/3' case
on my end. v14 was taking 8000ms but v15 took ~9100ms. If we add the
tmp_hit_eof variable then the regression disappears. Also, if I use a
struct like below, regression disappears again.
typedef struct CopyReadLineSIMDResult
{
int input_buf_ptr;
bool hit_eof;
bool result;
} CopyReadLineSIMDResult;
When I removed the tmp_hit_eof variable on v14, I didn't encounter any
regression. I really don't understand why this is happening on my end.
Manni didn't encounter any regression on the benchmark [1].
I benchmarked v15 and both of the cases above:
------------------------------------------------------------
Results for default_toast_compression = 'lz4':
+--------------------------------------------------+
| Optimization: -O2 |
+-------------------+--------------+---------------+
| | Text | CSV |
+-------------------+------+-------+-------+-------+
| WIDE | None | 1/3 | None | 1/3 |
+-------------------+------+-------+-------+-------+
| Old master | 4260 | 4789 | 5930 | 8276 |
+-------------------+------+-------+-------+-------+
| v14 | 2489 | 4439 | 2529 | 8098 |
+-------------------+------+-------+-------+-------+
| v15 | 2494 | 4235 | 2490 | 9140 |
+-------------------+------+-------+-------+-------+
| v15 + tmp_hit_eof | 2487 | 4539 | 2478 | 8041 |
+-------------------+------+-------+-------+-------+
| v15 + struct | 2490 | 4531 | 2483 | 7756 |
+-------------------+------+-------+-------+-------+
| | | | | |
+-------------------+------+-------+-------+-------+
| | | | | |
+-------------------+------+-------+-------+-------+
| | Text | CSV |
+-------------------+------+-------+-------+-------+
| NARROW | None | 1/3 | None | 1/3 |
+-------------------+------+-------+-------+-------+
| Old master | 9955 | 10056 | 10329 | 10872 |
+-------------------+------+-------+-------+-------+
| v14 | 9917 | 10080 | 10104 | 10510 |
+-------------------+------+-------+-------+-------+
| v15 | 9898 | 10062 | 10232 | 10483 |
+-------------------+------+-------+-------+-------+
| v15 + tmp_hit_eof | 9847 | 10004 | 10192 | 10437 |
+-------------------+------+-------+-------+-------+
| v15 + struct | 9877 | 10008 | 10107 | 10521 |
+-------------------+------+-------+-------+-------+
------------------------------------------------------------
Results for default_toast_compression = 'pglz':
+---------------------------------------------------+
| Optimization: -O2 |
+-------------------+---------------+---------------+
| | Text | CSV |
+-------------------+-------+-------+-------+-------+
| WIDE | None | 1/3 | None | 1/3 |
+-------------------+-------+-------+-------+-------+
| Old master | 10579 | 10927 | 12276 | 14488 |
+-------------------+-------+-------+-------+-------+
| v14 | 8832 | 10646 | 8815 | 14352 |
+-------------------+-------+-------+-------+-------+
| v15 | 8859 | 10489 | 8835 | 15414 |
+-------------------+-------+-------+-------+-------+
| v15 + tmp_hit_eof | 8828 | 10829 | 8840 | 14297 |
+-------------------+-------+-------+-------+-------+
| v15 + struct | 8847 | 10829 | 8846 | 14003 |
+-------------------+-------+-------+-------+-------+
| | | | | |
+-------------------+-------+-------+-------+-------+
| | | | | |
+-------------------+-------+-------+-------+-------+
| | Text | CSV |
+-------------------+-------+-------+-------+-------+
| NARROW | None | 1/3 | None | 1/3 |
+-------------------+-------+-------+-------+-------+
| Old master | 9952 | 10342 | 10112 | 10861 |
+-------------------+-------+-------+-------+-------+
| v14 | 9907 | 10344 | 10103 | 10492 |
+-------------------+-------+-------+-------+-------+
| v15 | 9897 | 10261 | 10126 | 10490 |
+-------------------+-------+-------+-------+-------+
| v15 + tmp_hit_eof | 9848 | 10218 | 10184 | 10425 |
+-------------------+-------+-------+-------+-------+
| v15 + struct | 9858 | 10150 | 10116 | 10464 |
+-------------------+-------+-------+-------+-------+
------------------------------------------------------------
It can be seen that the 'csv & wide & 1/3' case is much better on 'v15
+ struct' and 'v15 + tmp_hit_eof' but 'text & wide & 1/3' case is a
bit worse but still better than master.
Regardless of the issues above, I encountered a compiler warning on
the v15, if 'USE_NO_SIMD' is defined, then this warning appears:
copyfromparse.c:1780:1: warning: label ‘out’ defined but not used
[-Wunused-label]
Rest of the changes look good to me. v16 is attached, it fixes the
warning by protecting 'out' with '#ifndef USE_NO_SIMD', no other
changes. In addition to that, I put 'using CopyReadLineSIMDResult
struct' as a 0002 to get an opinion.
[1] https://postgr.es/m/CAKWEB6pMbdMDvhfaX1Z0eSULVQFYhEhssaRHdOxAX_5OYubxKw%40mail.gmail.com
--
Regards,
Nazir Bilal Yavuz
Microsoft
Attachments:
[text/x-patch] v16-0001-Optimize-COPY-FROM-FORMAT-text-csv-using-SIMD.patch (8.9K, ../../CAN55FZ0oYuVHZ=TdJofj0WA3mG77XumVsGJgG8gnav9kexr+wg@mail.gmail.com/2-v16-0001-Optimize-COPY-FROM-FORMAT-text-csv-using-SIMD.patch)
download | inline diff:
From 49e82abfc752032fb10e2c144f7656f6fdf78366 Mon Sep 17 00:00:00 2001
From: Nathan Bossart <nathan@postgresql.org>
Date: Thu, 12 Mar 2026 12:32:23 -0500
Subject: [PATCH v16 1/2] Optimize COPY FROM (FORMAT {text,csv}) using SIMD.
Presently, such commands scan the input buffer one byte at a time
looking for special characters. This commit adds a new path that
uses SIMD instructions to skip over chunks of data without any
special characters. This can be much faster.
To avoid regressions, SIMD processing is disabled for the remainder
of the COPY FROM command as soon as we encounter a short line or a
special character (except for end-of-line characters, else we'd
always disable it after the first line). This is perhaps too
conservative, but it could probably be made more lenient in the
future via fine-tuned heuristics.
Author: Nazir Bilal Yavuz <byavuz81@gmail.com>
Co-authored-by: Shinya Kato <shinya11.kato@gmail.com>
Reviewed-by: Ayoub Kazar <ma_kazar@esi.dz>
Reviewed-by: Andrew Dunstan <andrew@dunslane.net>
Reviewed-by: Neil Conway <neil.conway@gmail.com>
Tested-by: Manni Wood <manni.wood@enterprisedb.com>
Tested-by: Mark Wong <markwkm@gmail.com>
Discussion: https://postgr.es/m/CAOzEurSW8cNr6TPKsjrstnPfhf4QyQqB4tnPXGGe8N4e_v7Jig%40mail.gmail.com
---
src/backend/commands/copyfrom.c | 1 +
src/backend/commands/copyfromparse.c | 182 ++++++++++++++++++++++-
src/include/commands/copyfrom_internal.h | 1 +
3 files changed, 181 insertions(+), 3 deletions(-)
diff --git a/src/backend/commands/copyfrom.c b/src/backend/commands/copyfrom.c
index 0ece40557c8..95f6cb416a9 100644
--- a/src/backend/commands/copyfrom.c
+++ b/src/backend/commands/copyfrom.c
@@ -1746,6 +1746,7 @@ BeginCopyFrom(ParseState *pstate,
cstate->cur_attname = NULL;
cstate->cur_attval = NULL;
cstate->relname_only = false;
+ cstate->simd_enabled = true;
/*
* Allocate buffers for the input pipeline.
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index 84c8809a889..bae3bf6fb0d 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -72,6 +72,7 @@
#include "miscadmin.h"
#include "pgstat.h"
#include "port/pg_bswap.h"
+#include "port/simd.h"
#include "utils/builtins.h"
#include "utils/rel.h"
#include "utils/wait_event.h"
@@ -1311,6 +1312,152 @@ CopyReadLine(CopyFromState cstate, bool is_csv)
return result;
}
+#ifndef USE_NO_SIMD
+/*
+ * Helper function for CopyReadLineText() that uses SIMD instructions to scan
+ * the input buffer for special characters. This can be much faster.
+ *
+ * Note that we disable SIMD for the remainder of the COPY FROM command upon
+ * encountering a special character (except for end-of-line characters) or a
+ * short line. This is perhaps too conservative, but it should help avoid
+ * regressions. It could probably be made more lenient in the future via
+ * fine-tuned heuristics.
+ */
+static bool
+CopyReadLineTextSIMDHelper(CopyFromState cstate, bool is_csv,
+ bool *hit_eof_p, int *input_buf_ptr_p)
+{
+ char *copy_input_buf;
+ int input_buf_ptr;
+ int copy_buf_len;
+ bool unique_esc_char; /* for csv, do quote/esc chars differ? */
+ bool first = true;
+ bool result = false;
+ const Vector8 nl_vec = vector8_broadcast('\n');
+ const Vector8 cr_vec = vector8_broadcast('\r');
+ Vector8 bs_or_quote_vec; /* '\' for text, quote for csv */
+ Vector8 esc_vec; /* only for csv */
+
+ if (is_csv)
+ {
+ char quote = cstate->opts.quote[0];
+ char esc = cstate->opts.escape[0];
+
+ bs_or_quote_vec = vector8_broadcast(quote);
+ esc_vec = vector8_broadcast(esc);
+ unique_esc_char = (quote != esc);
+ }
+ else
+ {
+ bs_or_quote_vec = vector8_broadcast('\\');
+ unique_esc_char = false;
+ }
+
+ /*
+ * For a little extra speed within the loop, we copy some state members
+ * into local variables. Note that we need to use a separate local
+ * variable for input_buf_ptr so that the REFILL_LINEBUF macro works. We
+ * copy its value into the input_buf_ptr_p argument before returning.
+ */
+ copy_input_buf = cstate->input_buf;
+ input_buf_ptr = cstate->input_buf_index;
+ copy_buf_len = cstate->input_buf_len;
+
+ /*
+ * See the corresponding loop in CopyReadLineText() for more information
+ * about the purpose of this loop. This one does the same thing using
+ * SIMD instructions, although we are quick to bail out to the scalar path
+ * if we encounter a special character.
+ */
+ for (;;)
+ {
+ Vector8 chunk;
+ Vector8 match;
+
+ /* Load more data if needed. */
+ if (copy_buf_len - input_buf_ptr < sizeof(Vector8))
+ {
+ REFILL_LINEBUF;
+
+ CopyLoadInputBuf(cstate);
+ /* update our local variables */
+ *hit_eof_p = cstate->input_reached_eof;
+ input_buf_ptr = cstate->input_buf_index;
+ copy_buf_len = cstate->input_buf_len;
+
+ /*
+ * If we are completely out of data, break out of the loop,
+ * reporting EOF.
+ */
+ if (INPUT_BUF_BYTES(cstate) <= 0)
+ {
+ result = true;
+ break;
+ }
+ }
+
+ /*
+ * If we still don't have enough data for the SIMD path, fall back to
+ * the scalar code. Note that this doesn't necessarily mean we
+ * encountered a short line, so we leave cstate->simd_enabled set to
+ * true.
+ */
+ if (copy_buf_len - input_buf_ptr < sizeof(Vector8))
+ break;
+
+ /*
+ * If we made it here, we have at least enough data to fit in a
+ * Vector8, so we can use SIMD instructions to scan for special
+ * characters.
+ */
+ vector8_load(&chunk, (const uint8 *) ©_input_buf[input_buf_ptr]);
+
+ /*
+ * Check for \n, \r, \\ (for text), quotes (for csv), and escapes (for
+ * csv, if different from quotes).
+ */
+ match = vector8_eq(chunk, nl_vec);
+ match = vector8_or(match, vector8_eq(chunk, cr_vec));
+ match = vector8_or(match, vector8_eq(chunk, bs_or_quote_vec));
+ if (unique_esc_char)
+ match = vector8_or(match, vector8_eq(chunk, esc_vec));
+
+ /*
+ * If we found a special character, advance to it and hand off to the
+ * scalar path. Except for end-of-line characters, we also disable
+ * SIMD processing for the remainder of the COPY FROM command.
+ */
+ if (vector8_is_highbit_set(match))
+ {
+ uint32 mask;
+ char c;
+
+ mask = vector8_highbit_mask(match);
+ input_buf_ptr += pg_rightmost_one_pos32(mask);
+
+ /*
+ * Don't disable SIMD if we found \n or \r, else we'd stop using
+ * SIMD instructions after the first line. As an exception, we do
+ * disable it if this is the first vector we processed, as that
+ * means the line is too short for SIMD.
+ */
+ c = copy_input_buf[input_buf_ptr];
+ if (first || (c != '\n' && c != '\r'))
+ cstate->simd_enabled = false;
+
+ break;
+ }
+
+ /* That chunk was clear of special characters, so we can skip it. */
+ input_buf_ptr += sizeof(Vector8);
+ first = false;
+ }
+
+ *input_buf_ptr_p = input_buf_ptr;
+ return result;
+}
+#endif /* ! USE_NO_SIMD */
+
/*
* CopyReadLineText - inner loop of CopyReadLine for text mode
*/
@@ -1361,11 +1508,36 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
* input_buf_ptr have been determined to be part of the line, but not yet
* transferred to line_buf.
*
- * For a little extra speed within the loop, we copy input_buf and
- * input_buf_len into local variables.
+ * For a little extra speed within the loop, we copy some state
+ * information into local variables. input_buf_ptr could be changed in
+ * the SIMD path, so we must set that one before it. The others are set
+ * afterwards.
*/
- copy_input_buf = cstate->input_buf;
input_buf_ptr = cstate->input_buf_index;
+
+ /*
+ * We first try to use SIMD for the task described above, falling back to
+ * the scalar path (i.e., the loop below) if needed.
+ */
+#ifndef USE_NO_SIMD
+ if (cstate->simd_enabled)
+ {
+ /*
+ * Using a temporary variable seems to encourage the compiler to keep
+ * it in a register, which is beneficial for performance.
+ */
+ int tmp_input_buf_ptr;
+
+ result = CopyReadLineTextSIMDHelper(cstate, is_csv, &hit_eof,
+ &tmp_input_buf_ptr);
+ input_buf_ptr = tmp_input_buf_ptr;
+
+ if (result)
+ goto out;
+ }
+#endif /* ! USE_NO_SIMD */
+
+ copy_input_buf = cstate->input_buf;
copy_buf_len = cstate->input_buf_len;
for (;;)
@@ -1605,6 +1777,10 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
}
} /* end of outer loop */
+#ifndef USE_NO_SIMD
+out:
+#endif /* ! USE_NO_SIMD */
+
/*
* Transfer any still-uncopied data to line_buf.
*/
diff --git a/src/include/commands/copyfrom_internal.h b/src/include/commands/copyfrom_internal.h
index f892c343157..9d3e244ee55 100644
--- a/src/include/commands/copyfrom_internal.h
+++ b/src/include/commands/copyfrom_internal.h
@@ -108,6 +108,7 @@ typedef struct CopyFromStateData
* att */
bool *defaults; /* if DEFAULT marker was found for
* corresponding att */
+ bool simd_enabled; /* use SIMD to scan for special chars? */
/*
* True if the corresponding attribute's is a constrained domain. This
--
2.47.3
[text/x-patch] v16-0002-Use-CopyReadLineSIMDResult-struct.patch (4.3K, ../../CAN55FZ0oYuVHZ=TdJofj0WA3mG77XumVsGJgG8gnav9kexr+wg@mail.gmail.com/3-v16-0002-Use-CopyReadLineSIMDResult-struct.patch)
download | inline diff:
From a32d853e020b1660510f960e7ba52707bbd6afe3 Mon Sep 17 00:00:00 2001
From: Nazir Bilal Yavuz <byavuz81@gmail.com>
Date: Fri, 13 Mar 2026 14:25:45 +0300
Subject: [PATCH v16 2/2] Use CopyReadLineSIMDResult struct
---
src/backend/commands/copyfromparse.c | 44 +++++++++++++++++-----------
src/tools/pgindent/typedefs.list | 1 +
2 files changed, 28 insertions(+), 17 deletions(-)
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index bae3bf6fb0d..3e3358af9e0 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -1313,6 +1313,17 @@ CopyReadLine(CopyFromState cstate, bool is_csv)
}
#ifndef USE_NO_SIMD
+/*
+ * Result of CopyReadLineTextSIMDHelper, returned by value to avoid
+ * pointer parameters that could inhibit register allocation in the caller.
+ */
+typedef struct CopyReadLineSIMDResult
+{
+ int input_buf_ptr;
+ bool hit_eof;
+ bool result;
+} CopyReadLineSIMDResult;
+
/*
* Helper function for CopyReadLineText() that uses SIMD instructions to scan
* the input buffer for special characters. This can be much faster.
@@ -1323,21 +1334,23 @@ CopyReadLine(CopyFromState cstate, bool is_csv)
* regressions. It could probably be made more lenient in the future via
* fine-tuned heuristics.
*/
-static bool
-CopyReadLineTextSIMDHelper(CopyFromState cstate, bool is_csv,
- bool *hit_eof_p, int *input_buf_ptr_p)
+static CopyReadLineSIMDResult
+CopyReadLineTextSIMDHelper(CopyFromState cstate, bool is_csv)
{
+ CopyReadLineSIMDResult ret;
char *copy_input_buf;
int input_buf_ptr;
int copy_buf_len;
bool unique_esc_char; /* for csv, do quote/esc chars differ? */
bool first = true;
- bool result = false;
const Vector8 nl_vec = vector8_broadcast('\n');
const Vector8 cr_vec = vector8_broadcast('\r');
Vector8 bs_or_quote_vec; /* '\' for text, quote for csv */
Vector8 esc_vec; /* only for csv */
+ ret.hit_eof = false;
+ ret.result = false;
+
if (is_csv)
{
char quote = cstate->opts.quote[0];
@@ -1357,7 +1370,7 @@ CopyReadLineTextSIMDHelper(CopyFromState cstate, bool is_csv,
* For a little extra speed within the loop, we copy some state members
* into local variables. Note that we need to use a separate local
* variable for input_buf_ptr so that the REFILL_LINEBUF macro works. We
- * copy its value into the input_buf_ptr_p argument before returning.
+ * copy its value into the return struct before returning.
*/
copy_input_buf = cstate->input_buf;
input_buf_ptr = cstate->input_buf_index;
@@ -1381,7 +1394,7 @@ CopyReadLineTextSIMDHelper(CopyFromState cstate, bool is_csv,
CopyLoadInputBuf(cstate);
/* update our local variables */
- *hit_eof_p = cstate->input_reached_eof;
+ ret.hit_eof = cstate->input_reached_eof;
input_buf_ptr = cstate->input_buf_index;
copy_buf_len = cstate->input_buf_len;
@@ -1391,7 +1404,7 @@ CopyReadLineTextSIMDHelper(CopyFromState cstate, bool is_csv,
*/
if (INPUT_BUF_BYTES(cstate) <= 0)
{
- result = true;
+ ret.result = true;
break;
}
}
@@ -1453,8 +1466,8 @@ CopyReadLineTextSIMDHelper(CopyFromState cstate, bool is_csv,
first = false;
}
- *input_buf_ptr_p = input_buf_ptr;
- return result;
+ ret.input_buf_ptr = input_buf_ptr;
+ return ret;
}
#endif /* ! USE_NO_SIMD */
@@ -1522,15 +1535,12 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
#ifndef USE_NO_SIMD
if (cstate->simd_enabled)
{
- /*
- * Using a temporary variable seems to encourage the compiler to keep
- * it in a register, which is beneficial for performance.
- */
- int tmp_input_buf_ptr;
+ CopyReadLineSIMDResult simd_result;
- result = CopyReadLineTextSIMDHelper(cstate, is_csv, &hit_eof,
- &tmp_input_buf_ptr);
- input_buf_ptr = tmp_input_buf_ptr;
+ simd_result = CopyReadLineTextSIMDHelper(cstate, is_csv);
+ hit_eof = simd_result.hit_eof;
+ input_buf_ptr = simd_result.input_buf_ptr;
+ result = simd_result.result;
if (result)
goto out;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 0de55183793..2acc40533c6 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -538,6 +538,7 @@ CopyMethod
CopyMultiInsertBuffer
CopyMultiInsertInfo
CopyOnErrorChoice
+CopyReadLineSIMDResult
CopySeqResult
CopySource
CopyStmt
--
2.47.3
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 22:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-10 11:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-21 20:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Neil Conway <neil.conway@gmail.com>
2026-01-31 16:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 13:51 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 18:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 21:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-06 22:19 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 22:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-11 13:27 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-11 22:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-13 11:45 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-13 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 13:38 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-18 21:26 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 21:54 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 04:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-19 09:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 22:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-20 00:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-20 09:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-20 18:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-23 09:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 04:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-24 13:57 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 17:48 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-25 14:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-02 19:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-04 15:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-09 18:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-10 12:35 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-10 17:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 11:36 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 18:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 18:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 19:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 19:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 20:42 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-12 10:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-12 17:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-13 11:57 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
@ 2026-03-13 13:34 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-13 14:05 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
0 siblings, 1 reply; 178+ messages in thread
From: Nazir Bilal Yavuz @ 2026-03-13 13:34 UTC (permalink / raw)
To: Nathan Bossart <nathandbossart@gmail.com>; +Cc: Manni Wood <manni.wood@enterprisedb.com>; KAZAR Ayoub <ma_kazar@esi.dz>; Neil Conway <neil.conway@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
Hi,
On Fri, 13 Mar 2026 at 14:57, Nazir Bilal Yavuz <byavuz81@gmail.com> wrote:
>
> Unfortunately, v15 causes a regression for a 'csv & wide & 1/3' case
> on my end. v14 was taking 8000ms but v15 took ~9100ms. If we add the
> tmp_hit_eof variable then the regression disappears. Also, if I use a
> struct like below, regression disappears again.
> When I removed the tmp_hit_eof variable on v14, I didn't encounter any
> regression. I really don't understand why this is happening on my end.
> Manni didn't encounter any regression on the benchmark [1].
Problem might be related to gcc. I am using Debian Trixie and my
current gcc version is 'gcc version 14.2.0 (Debian 14.2.0-19)'. If I
compile Postgres with 'Debian clang version 19.1.7 (3+b1)', then there
is no regression, which makes more sense IMO.
Here is a comparison for csv & wide & 1/3 case. Postgres is compiled
with buildtype=debugoptimized and default_toast_compression is lz4.
+--------------------------------+
| CSV & WIDE & 1/3, LZ4, -O2 |
+--------------+--------+--------+
| | gcc | clang |
| | 14.0.2 | 19.1.7 |
+--------------+--------+--------+
| old master | 8250 | 10400 |
+--------------+--------+--------+
| v14 | 8100 | 9800 |
+--------------+--------+--------+
| v15 | 9200 | 9800 |
+--------------+--------+--------+
| v15 + struct | 7750 | 9800 |
+--------------+--------+--------+
--
Regards,
Nazir Bilal Yavuz
Microsoft
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 22:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-10 11:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-21 20:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Neil Conway <neil.conway@gmail.com>
2026-01-31 16:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 13:51 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 18:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 21:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-06 22:19 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 22:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-11 13:27 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-11 22:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-13 11:45 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-13 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 13:38 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-18 21:26 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 21:54 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 04:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-19 09:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 22:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-20 00:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-20 09:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-20 18:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-23 09:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 04:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-24 13:57 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 17:48 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-25 14:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-02 19:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-04 15:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-09 18:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-10 12:35 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-10 17:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 11:36 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 18:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 18:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 19:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 19:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 20:42 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-12 10:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-12 17:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-13 11:57 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-13 13:34 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
@ 2026-03-13 14:05 ` Nathan Bossart <nathandbossart@gmail.com>
2026-03-13 15:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Greg Burd <greg@burd.me>
2026-03-13 15:58 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
0 siblings, 2 replies; 178+ messages in thread
From: Nathan Bossart @ 2026-03-13 14:05 UTC (permalink / raw)
To: Nazir Bilal Yavuz <byavuz81@gmail.com>; +Cc: Manni Wood <manni.wood@enterprisedb.com>; KAZAR Ayoub <ma_kazar@esi.dz>; Neil Conway <neil.conway@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
On Fri, Mar 13, 2026 at 04:34:49PM +0300, Nazir Bilal Yavuz wrote:
> On Fri, 13 Mar 2026 at 14:57, Nazir Bilal Yavuz <byavuz81@gmail.com> wrote:
>> Unfortunately, v15 causes a regression for a 'csv & wide & 1/3' case
>> on my end. v14 was taking 8000ms but v15 took ~9100ms. If we add the
>> tmp_hit_eof variable then the regression disappears. Also, if I use a
>> struct like below, regression disappears again.
>
>> When I removed the tmp_hit_eof variable on v14, I didn't encounter any
>> regression. I really don't understand why this is happening on my end.
>> Manni didn't encounter any regression on the benchmark [1].
>
> Problem might be related to gcc. I am using Debian Trixie and my
> current gcc version is 'gcc version 14.2.0 (Debian 14.2.0-19)'. If I
> compile Postgres with 'Debian clang version 19.1.7 (3+b1)', then there
> is no regression, which makes more sense IMO.
Let's just re-add the temporary variable for hit_eof. The struct idea is
clever, but it's just a little more complicated than I think is necessary
here.
I've also removed the goto in favor of just duplicating the "out" code,
like you had before. I'd like to avoid sporadic #ifndef USE_NO_SIMD uses,
and goto is out of fashion, anyway.
--
nathan
From cdb600ff363a583e53594e03f459e3ab3eea579e Mon Sep 17 00:00:00 2001
From: Nathan Bossart <nathan@postgresql.org>
Date: Thu, 12 Mar 2026 12:32:23 -0500
Subject: [PATCH v17 1/1] Optimize COPY FROM (FORMAT {text,csv}) using SIMD.
Presently, such commands scan the input buffer one byte at a time
looking for special characters. This commit adds a new path that
uses SIMD instructions to skip over chunks of data without any
special characters. This can be much faster.
To avoid regressions, SIMD processing is disabled for the remainder
of the COPY FROM command as soon as we encounter a short line or a
special character (except for end-of-line characters, else we'd
always disable it after the first line). This is perhaps too
conservative, but it could probably be made more lenient in the
future via fine-tuned heuristics.
Author: Nazir Bilal Yavuz <byavuz81@gmail.com>
Co-authored-by: Shinya Kato <shinya11.kato@gmail.com>
Reviewed-by: Ayoub Kazar <ma_kazar@esi.dz>
Reviewed-by: Andrew Dunstan <andrew@dunslane.net>
Reviewed-by: Neil Conway <neil.conway@gmail.com>
Tested-by: Manni Wood <manni.wood@enterprisedb.com>
Tested-by: Mark Wong <markwkm@gmail.com>
Discussion: https://postgr.es/m/CAOzEurSW8cNr6TPKsjrstnPfhf4QyQqB4tnPXGGe8N4e_v7Jig%40mail.gmail.com
---
src/backend/commands/copyfrom.c | 1 +
src/backend/commands/copyfromparse.c | 185 ++++++++++++++++++++++-
src/include/commands/copyfrom_internal.h | 1 +
3 files changed, 184 insertions(+), 3 deletions(-)
diff --git a/src/backend/commands/copyfrom.c b/src/backend/commands/copyfrom.c
index 0ece40557c8..95f6cb416a9 100644
--- a/src/backend/commands/copyfrom.c
+++ b/src/backend/commands/copyfrom.c
@@ -1746,6 +1746,7 @@ BeginCopyFrom(ParseState *pstate,
cstate->cur_attname = NULL;
cstate->cur_attval = NULL;
cstate->relname_only = false;
+ cstate->simd_enabled = true;
/*
* Allocate buffers for the input pipeline.
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index 84c8809a889..00ee4154b8b 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -72,6 +72,7 @@
#include "miscadmin.h"
#include "pgstat.h"
#include "port/pg_bswap.h"
+#include "port/simd.h"
#include "utils/builtins.h"
#include "utils/rel.h"
#include "utils/wait_event.h"
@@ -1311,6 +1312,152 @@ CopyReadLine(CopyFromState cstate, bool is_csv)
return result;
}
+#ifndef USE_NO_SIMD
+/*
+ * Helper function for CopyReadLineText() that uses SIMD instructions to scan
+ * the input buffer for special characters. This can be much faster.
+ *
+ * Note that we disable SIMD for the remainder of the COPY FROM command upon
+ * encountering a special character (except for end-of-line characters) or a
+ * short line. This is perhaps too conservative, but it should help avoid
+ * regressions. It could probably be made more lenient in the future via
+ * fine-tuned heuristics.
+ */
+static bool
+CopyReadLineTextSIMDHelper(CopyFromState cstate, bool is_csv,
+ bool *hit_eof_p, int *input_buf_ptr_p)
+{
+ char *copy_input_buf;
+ int input_buf_ptr;
+ int copy_buf_len;
+ bool unique_esc_char; /* for csv, do quote/esc chars differ? */
+ bool first = true;
+ bool result = false;
+ const Vector8 nl_vec = vector8_broadcast('\n');
+ const Vector8 cr_vec = vector8_broadcast('\r');
+ Vector8 bs_or_quote_vec; /* '\' for text, quote for csv */
+ Vector8 esc_vec; /* only for csv */
+
+ if (is_csv)
+ {
+ char quote = cstate->opts.quote[0];
+ char esc = cstate->opts.escape[0];
+
+ bs_or_quote_vec = vector8_broadcast(quote);
+ esc_vec = vector8_broadcast(esc);
+ unique_esc_char = (quote != esc);
+ }
+ else
+ {
+ bs_or_quote_vec = vector8_broadcast('\\');
+ unique_esc_char = false;
+ }
+
+ /*
+ * For a little extra speed within the loop, we copy some state members
+ * into local variables. Note that we need to use a separate local
+ * variable for input_buf_ptr so that the REFILL_LINEBUF macro works. We
+ * copy its value into the input_buf_ptr_p argument before returning.
+ */
+ copy_input_buf = cstate->input_buf;
+ input_buf_ptr = cstate->input_buf_index;
+ copy_buf_len = cstate->input_buf_len;
+
+ /*
+ * See the corresponding loop in CopyReadLineText() for more information
+ * about the purpose of this loop. This one does the same thing using
+ * SIMD instructions, although we are quick to bail out to the scalar path
+ * if we encounter a special character.
+ */
+ for (;;)
+ {
+ Vector8 chunk;
+ Vector8 match;
+
+ /* Load more data if needed. */
+ if (copy_buf_len - input_buf_ptr < sizeof(Vector8))
+ {
+ REFILL_LINEBUF;
+
+ CopyLoadInputBuf(cstate);
+ /* update our local variables */
+ *hit_eof_p = cstate->input_reached_eof;
+ input_buf_ptr = cstate->input_buf_index;
+ copy_buf_len = cstate->input_buf_len;
+
+ /*
+ * If we are completely out of data, break out of the loop,
+ * reporting EOF.
+ */
+ if (INPUT_BUF_BYTES(cstate) <= 0)
+ {
+ result = true;
+ break;
+ }
+ }
+
+ /*
+ * If we still don't have enough data for the SIMD path, fall back to
+ * the scalar code. Note that this doesn't necessarily mean we
+ * encountered a short line, so we leave cstate->simd_enabled set to
+ * true.
+ */
+ if (copy_buf_len - input_buf_ptr < sizeof(Vector8))
+ break;
+
+ /*
+ * If we made it here, we have at least enough data to fit in a
+ * Vector8, so we can use SIMD instructions to scan for special
+ * characters.
+ */
+ vector8_load(&chunk, (const uint8 *) ©_input_buf[input_buf_ptr]);
+
+ /*
+ * Check for \n, \r, \\ (for text), quotes (for csv), and escapes (for
+ * csv, if different from quotes).
+ */
+ match = vector8_eq(chunk, nl_vec);
+ match = vector8_or(match, vector8_eq(chunk, cr_vec));
+ match = vector8_or(match, vector8_eq(chunk, bs_or_quote_vec));
+ if (unique_esc_char)
+ match = vector8_or(match, vector8_eq(chunk, esc_vec));
+
+ /*
+ * If we found a special character, advance to it and hand off to the
+ * scalar path. Except for end-of-line characters, we also disable
+ * SIMD processing for the remainder of the COPY FROM command.
+ */
+ if (vector8_is_highbit_set(match))
+ {
+ uint32 mask;
+ char c;
+
+ mask = vector8_highbit_mask(match);
+ input_buf_ptr += pg_rightmost_one_pos32(mask);
+
+ /*
+ * Don't disable SIMD if we found \n or \r, else we'd stop using
+ * SIMD instructions after the first line. As an exception, we do
+ * disable it if this is the first vector we processed, as that
+ * means the line is too short for SIMD.
+ */
+ c = copy_input_buf[input_buf_ptr];
+ if (first || (c != '\n' && c != '\r'))
+ cstate->simd_enabled = false;
+
+ break;
+ }
+
+ /* That chunk was clear of special characters, so we can skip it. */
+ input_buf_ptr += sizeof(Vector8);
+ first = false;
+ }
+
+ *input_buf_ptr_p = input_buf_ptr;
+ return result;
+}
+#endif /* ! USE_NO_SIMD */
+
/*
* CopyReadLineText - inner loop of CopyReadLine for text mode
*/
@@ -1361,11 +1508,43 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
* input_buf_ptr have been determined to be part of the line, but not yet
* transferred to line_buf.
*
- * For a little extra speed within the loop, we copy input_buf and
- * input_buf_len into local variables.
+ * For a little extra speed within the loop, we copy some state
+ * information into local variables. input_buf_ptr could be changed in
+ * the SIMD path, so we must set that one before it. The others are set
+ * afterwards.
*/
- copy_input_buf = cstate->input_buf;
input_buf_ptr = cstate->input_buf_index;
+
+ /*
+ * We first try to use SIMD for the task described above, falling back to
+ * the scalar path (i.e., the loop below) if needed.
+ */
+#ifndef USE_NO_SIMD
+ if (cstate->simd_enabled)
+ {
+ /*
+ * Using temporary variables seems to encourage the compiler to keep
+ * them in a register, which is beneficial for performance.
+ */
+ bool tmp_hit_eof = false;
+ int tmp_input_buf_ptr = 0; /* silence compiler warning */
+
+ result = CopyReadLineTextSIMDHelper(cstate, is_csv, &tmp_hit_eof,
+ &tmp_input_buf_ptr);
+ hit_eof = tmp_hit_eof;
+ input_buf_ptr = tmp_input_buf_ptr;
+
+ if (result)
+ {
+ /* Transfer any still-uncopied data to line_buf. */
+ REFILL_LINEBUF;
+
+ return result;
+ }
+ }
+#endif /* ! USE_NO_SIMD */
+
+ copy_input_buf = cstate->input_buf;
copy_buf_len = cstate->input_buf_len;
for (;;)
diff --git a/src/include/commands/copyfrom_internal.h b/src/include/commands/copyfrom_internal.h
index f892c343157..9d3e244ee55 100644
--- a/src/include/commands/copyfrom_internal.h
+++ b/src/include/commands/copyfrom_internal.h
@@ -108,6 +108,7 @@ typedef struct CopyFromStateData
* att */
bool *defaults; /* if DEFAULT marker was found for
* corresponding att */
+ bool simd_enabled; /* use SIMD to scan for special chars? */
/*
* True if the corresponding attribute's is a constrained domain. This
--
2.50.1 (Apple Git-155)
Attachments:
[text/plain] v17-0001-Optimize-COPY-FROM-FORMAT-text-csv-using-SIMD.patch (8.9K, ../../abQZmZ0KZesCxfR0@nathan/2-v17-0001-Optimize-COPY-FROM-FORMAT-text-csv-using-SIMD.patch)
download | inline diff:
From cdb600ff363a583e53594e03f459e3ab3eea579e Mon Sep 17 00:00:00 2001
From: Nathan Bossart <nathan@postgresql.org>
Date: Thu, 12 Mar 2026 12:32:23 -0500
Subject: [PATCH v17 1/1] Optimize COPY FROM (FORMAT {text,csv}) using SIMD.
Presently, such commands scan the input buffer one byte at a time
looking for special characters. This commit adds a new path that
uses SIMD instructions to skip over chunks of data without any
special characters. This can be much faster.
To avoid regressions, SIMD processing is disabled for the remainder
of the COPY FROM command as soon as we encounter a short line or a
special character (except for end-of-line characters, else we'd
always disable it after the first line). This is perhaps too
conservative, but it could probably be made more lenient in the
future via fine-tuned heuristics.
Author: Nazir Bilal Yavuz <byavuz81@gmail.com>
Co-authored-by: Shinya Kato <shinya11.kato@gmail.com>
Reviewed-by: Ayoub Kazar <ma_kazar@esi.dz>
Reviewed-by: Andrew Dunstan <andrew@dunslane.net>
Reviewed-by: Neil Conway <neil.conway@gmail.com>
Tested-by: Manni Wood <manni.wood@enterprisedb.com>
Tested-by: Mark Wong <markwkm@gmail.com>
Discussion: https://postgr.es/m/CAOzEurSW8cNr6TPKsjrstnPfhf4QyQqB4tnPXGGe8N4e_v7Jig%40mail.gmail.com
---
src/backend/commands/copyfrom.c | 1 +
src/backend/commands/copyfromparse.c | 185 ++++++++++++++++++++++-
src/include/commands/copyfrom_internal.h | 1 +
3 files changed, 184 insertions(+), 3 deletions(-)
diff --git a/src/backend/commands/copyfrom.c b/src/backend/commands/copyfrom.c
index 0ece40557c8..95f6cb416a9 100644
--- a/src/backend/commands/copyfrom.c
+++ b/src/backend/commands/copyfrom.c
@@ -1746,6 +1746,7 @@ BeginCopyFrom(ParseState *pstate,
cstate->cur_attname = NULL;
cstate->cur_attval = NULL;
cstate->relname_only = false;
+ cstate->simd_enabled = true;
/*
* Allocate buffers for the input pipeline.
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index 84c8809a889..00ee4154b8b 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -72,6 +72,7 @@
#include "miscadmin.h"
#include "pgstat.h"
#include "port/pg_bswap.h"
+#include "port/simd.h"
#include "utils/builtins.h"
#include "utils/rel.h"
#include "utils/wait_event.h"
@@ -1311,6 +1312,152 @@ CopyReadLine(CopyFromState cstate, bool is_csv)
return result;
}
+#ifndef USE_NO_SIMD
+/*
+ * Helper function for CopyReadLineText() that uses SIMD instructions to scan
+ * the input buffer for special characters. This can be much faster.
+ *
+ * Note that we disable SIMD for the remainder of the COPY FROM command upon
+ * encountering a special character (except for end-of-line characters) or a
+ * short line. This is perhaps too conservative, but it should help avoid
+ * regressions. It could probably be made more lenient in the future via
+ * fine-tuned heuristics.
+ */
+static bool
+CopyReadLineTextSIMDHelper(CopyFromState cstate, bool is_csv,
+ bool *hit_eof_p, int *input_buf_ptr_p)
+{
+ char *copy_input_buf;
+ int input_buf_ptr;
+ int copy_buf_len;
+ bool unique_esc_char; /* for csv, do quote/esc chars differ? */
+ bool first = true;
+ bool result = false;
+ const Vector8 nl_vec = vector8_broadcast('\n');
+ const Vector8 cr_vec = vector8_broadcast('\r');
+ Vector8 bs_or_quote_vec; /* '\' for text, quote for csv */
+ Vector8 esc_vec; /* only for csv */
+
+ if (is_csv)
+ {
+ char quote = cstate->opts.quote[0];
+ char esc = cstate->opts.escape[0];
+
+ bs_or_quote_vec = vector8_broadcast(quote);
+ esc_vec = vector8_broadcast(esc);
+ unique_esc_char = (quote != esc);
+ }
+ else
+ {
+ bs_or_quote_vec = vector8_broadcast('\\');
+ unique_esc_char = false;
+ }
+
+ /*
+ * For a little extra speed within the loop, we copy some state members
+ * into local variables. Note that we need to use a separate local
+ * variable for input_buf_ptr so that the REFILL_LINEBUF macro works. We
+ * copy its value into the input_buf_ptr_p argument before returning.
+ */
+ copy_input_buf = cstate->input_buf;
+ input_buf_ptr = cstate->input_buf_index;
+ copy_buf_len = cstate->input_buf_len;
+
+ /*
+ * See the corresponding loop in CopyReadLineText() for more information
+ * about the purpose of this loop. This one does the same thing using
+ * SIMD instructions, although we are quick to bail out to the scalar path
+ * if we encounter a special character.
+ */
+ for (;;)
+ {
+ Vector8 chunk;
+ Vector8 match;
+
+ /* Load more data if needed. */
+ if (copy_buf_len - input_buf_ptr < sizeof(Vector8))
+ {
+ REFILL_LINEBUF;
+
+ CopyLoadInputBuf(cstate);
+ /* update our local variables */
+ *hit_eof_p = cstate->input_reached_eof;
+ input_buf_ptr = cstate->input_buf_index;
+ copy_buf_len = cstate->input_buf_len;
+
+ /*
+ * If we are completely out of data, break out of the loop,
+ * reporting EOF.
+ */
+ if (INPUT_BUF_BYTES(cstate) <= 0)
+ {
+ result = true;
+ break;
+ }
+ }
+
+ /*
+ * If we still don't have enough data for the SIMD path, fall back to
+ * the scalar code. Note that this doesn't necessarily mean we
+ * encountered a short line, so we leave cstate->simd_enabled set to
+ * true.
+ */
+ if (copy_buf_len - input_buf_ptr < sizeof(Vector8))
+ break;
+
+ /*
+ * If we made it here, we have at least enough data to fit in a
+ * Vector8, so we can use SIMD instructions to scan for special
+ * characters.
+ */
+ vector8_load(&chunk, (const uint8 *) ©_input_buf[input_buf_ptr]);
+
+ /*
+ * Check for \n, \r, \\ (for text), quotes (for csv), and escapes (for
+ * csv, if different from quotes).
+ */
+ match = vector8_eq(chunk, nl_vec);
+ match = vector8_or(match, vector8_eq(chunk, cr_vec));
+ match = vector8_or(match, vector8_eq(chunk, bs_or_quote_vec));
+ if (unique_esc_char)
+ match = vector8_or(match, vector8_eq(chunk, esc_vec));
+
+ /*
+ * If we found a special character, advance to it and hand off to the
+ * scalar path. Except for end-of-line characters, we also disable
+ * SIMD processing for the remainder of the COPY FROM command.
+ */
+ if (vector8_is_highbit_set(match))
+ {
+ uint32 mask;
+ char c;
+
+ mask = vector8_highbit_mask(match);
+ input_buf_ptr += pg_rightmost_one_pos32(mask);
+
+ /*
+ * Don't disable SIMD if we found \n or \r, else we'd stop using
+ * SIMD instructions after the first line. As an exception, we do
+ * disable it if this is the first vector we processed, as that
+ * means the line is too short for SIMD.
+ */
+ c = copy_input_buf[input_buf_ptr];
+ if (first || (c != '\n' && c != '\r'))
+ cstate->simd_enabled = false;
+
+ break;
+ }
+
+ /* That chunk was clear of special characters, so we can skip it. */
+ input_buf_ptr += sizeof(Vector8);
+ first = false;
+ }
+
+ *input_buf_ptr_p = input_buf_ptr;
+ return result;
+}
+#endif /* ! USE_NO_SIMD */
+
/*
* CopyReadLineText - inner loop of CopyReadLine for text mode
*/
@@ -1361,11 +1508,43 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
* input_buf_ptr have been determined to be part of the line, but not yet
* transferred to line_buf.
*
- * For a little extra speed within the loop, we copy input_buf and
- * input_buf_len into local variables.
+ * For a little extra speed within the loop, we copy some state
+ * information into local variables. input_buf_ptr could be changed in
+ * the SIMD path, so we must set that one before it. The others are set
+ * afterwards.
*/
- copy_input_buf = cstate->input_buf;
input_buf_ptr = cstate->input_buf_index;
+
+ /*
+ * We first try to use SIMD for the task described above, falling back to
+ * the scalar path (i.e., the loop below) if needed.
+ */
+#ifndef USE_NO_SIMD
+ if (cstate->simd_enabled)
+ {
+ /*
+ * Using temporary variables seems to encourage the compiler to keep
+ * them in a register, which is beneficial for performance.
+ */
+ bool tmp_hit_eof = false;
+ int tmp_input_buf_ptr = 0; /* silence compiler warning */
+
+ result = CopyReadLineTextSIMDHelper(cstate, is_csv, &tmp_hit_eof,
+ &tmp_input_buf_ptr);
+ hit_eof = tmp_hit_eof;
+ input_buf_ptr = tmp_input_buf_ptr;
+
+ if (result)
+ {
+ /* Transfer any still-uncopied data to line_buf. */
+ REFILL_LINEBUF;
+
+ return result;
+ }
+ }
+#endif /* ! USE_NO_SIMD */
+
+ copy_input_buf = cstate->input_buf;
copy_buf_len = cstate->input_buf_len;
for (;;)
diff --git a/src/include/commands/copyfrom_internal.h b/src/include/commands/copyfrom_internal.h
index f892c343157..9d3e244ee55 100644
--- a/src/include/commands/copyfrom_internal.h
+++ b/src/include/commands/copyfrom_internal.h
@@ -108,6 +108,7 @@ typedef struct CopyFromStateData
* att */
bool *defaults; /* if DEFAULT marker was found for
* corresponding att */
+ bool simd_enabled; /* use SIMD to scan for special chars? */
/*
* True if the corresponding attribute's is a constrained domain. This
--
2.50.1 (Apple Git-155)
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 22:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-10 11:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-21 20:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Neil Conway <neil.conway@gmail.com>
2026-01-31 16:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 13:51 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 18:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 21:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-06 22:19 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 22:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-11 13:27 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-11 22:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-13 11:45 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-13 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 13:38 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-18 21:26 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 21:54 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 04:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-19 09:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 22:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-20 00:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-20 09:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-20 18:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-23 09:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 04:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-24 13:57 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 17:48 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-25 14:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-02 19:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-04 15:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-09 18:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-10 12:35 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-10 17:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 11:36 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 18:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 18:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 19:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 19:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 20:42 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-12 10:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-12 17:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-13 11:57 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-13 13:34 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-13 14:05 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
@ 2026-03-13 15:29 ` Greg Burd <greg@burd.me>
2026-03-13 16:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
1 sibling, 1 reply; 178+ messages in thread
From: Greg Burd @ 2026-03-13 15:29 UTC (permalink / raw)
To: Nathan Bossart <nathandbossart@gmail.com>; Nazir Bilal Yavuz <byavuz81@gmail.com>; +Cc: Manni Wood <manni.wood@enterprisedb.com>; KAZAR Ayoub <ma_kazar@esi.dz>; Neil Conway <neil.conway@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
On Fri, Mar 13, 2026, at 10:05 AM, Nathan Bossart wrote:
> On Fri, Mar 13, 2026 at 04:34:49PM +0300, Nazir Bilal Yavuz wrote:
>> On Fri, 13 Mar 2026 at 14:57, Nazir Bilal Yavuz <byavuz81@gmail.com> wrote:
>>> Unfortunately, v15 causes a regression for a 'csv & wide & 1/3' case
>>> on my end. v14 was taking 8000ms but v15 took ~9100ms. If we add the
>>> tmp_hit_eof variable then the regression disappears. Also, if I use a
>>> struct like below, regression disappears again.
>>
>>> When I removed the tmp_hit_eof variable on v14, I didn't encounter any
>>> regression. I really don't understand why this is happening on my end.
>>> Manni didn't encounter any regression on the benchmark [1].
>>
>> Problem might be related to gcc. I am using Debian Trixie and my
>> current gcc version is 'gcc version 14.2.0 (Debian 14.2.0-19)'. If I
>> compile Postgres with 'Debian clang version 19.1.7 (3+b1)', then there
>> is no regression, which makes more sense IMO.
>
> Let's just re-add the temporary variable for hit_eof. The struct idea is
> clever, but it's just a little more complicated than I think is necessary
> here.
>
> I've also removed the goto in favor of just duplicating the "out" code,
> like you had before. I'd like to avoid sporadic #ifndef USE_NO_SIMD uses,
> and goto is out of fashion, anyway.
Hey Nathan, Nazir, et. al.,
I've always been a fan of these kinds of optimization so I couldn't resist reviewing, but I know you're ready to commit so I'll just check on some systems I have. :)
At first glance the implementation seems conservative, but correct and safe. Local testing on on Linux/FreeBSD x86_64, and Win11/aarch64/MSVC seem good. I also tried IllumOS/SPARCv9 and with some fixes (from another active thread) to the build system and it worked just fine too. I'm sure the 10 people care will be thrilled. ;-P
I ran into the "out:" label not defined because USE_NO_SIMD wasn't defined when I tested v15 on Linux/RISC-V, my build-farm animal "greenfly" which is an Orange Pi RV2 running Linux 24.04.4 LTS. Looks like changes this morning have fixed that.
I would have reported earlier this morning but I detoured into RISC-V specific fixes and optimizations. Interestingly, simd.h uses a fixed-width Vector8 type (16 bytes on SSE2/NEON) and relies on sizeof(Vector8) throughout, however RISC-V Vector types like vuint8m1_t are "sizeless" meaning that you cannot use sizeof() on them. To model that properly would require overhauling all of simd.h, which is way out of scope for this patch so I'll start a new thread for that (and CRC, and popcnt). I also created a few tests (attached) to check boundary conditions, I might add some along with the RISC-V work.
nice work, LGTM.
-greg
> --
> nathan
>
> Attachments:
> * v17-0001-Optimize-COPY-FROM-FORMAT-text-csv-using-SIMD.patch
Attachments:
[application/sql] test_simd_copy_boundaries.sql (7.2K, ../../43de48dc-701b-4735-881b-50bca6870f39@app.fastmail.com/2-test_simd_copy_boundaries.sql)
download
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 22:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-10 11:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-21 20:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Neil Conway <neil.conway@gmail.com>
2026-01-31 16:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 13:51 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 18:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 21:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-06 22:19 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 22:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-11 13:27 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-11 22:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-13 11:45 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-13 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 13:38 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-18 21:26 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 21:54 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 04:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-19 09:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 22:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-20 00:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-20 09:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-20 18:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-23 09:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 04:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-24 13:57 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 17:48 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-25 14:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-02 19:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-04 15:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-09 18:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-10 12:35 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-10 17:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 11:36 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 18:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 18:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 19:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 19:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 20:42 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-12 10:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-12 17:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-13 11:57 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-13 13:34 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-13 14:05 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-13 15:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Greg Burd <greg@burd.me>
@ 2026-03-13 16:14 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-13 16:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-13 17:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Greg Burd <greg@burd.me>
0 siblings, 2 replies; 178+ messages in thread
From: Nazir Bilal Yavuz @ 2026-03-13 16:14 UTC (permalink / raw)
To: Greg Burd <greg@burd.me>; +Cc: Nathan Bossart <nathandbossart@gmail.com>; Manni Wood <manni.wood@enterprisedb.com>; KAZAR Ayoub <ma_kazar@esi.dz>; Neil Conway <neil.conway@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
Hi Greg,
On Fri, 13 Mar 2026 at 18:29, Greg Burd <greg@burd.me> wrote:
>
> I've always been a fan of these kinds of optimization so I couldn't resist reviewing, but I know you're ready to commit so I'll just check on some systems I have. :)
Thank you for the review!
> At first glance the implementation seems conservative, but correct and safe. Local testing on on Linux/FreeBSD x86_64, and Win11/aarch64/MSVC seem good. I also tried IllumOS/SPARCv9 and with some fixes (from another active thread) to the build system and it worked just fine too. I'm sure the 10 people care will be thrilled. ;-
Yes, we can probably improve this further with heuristics, but for now
we wanted to avoid introducing any potential regressions.
> I also created a few tests (attached) to check boundary conditions, I might add some along with the RISC-V work.
Thank you for the tests! I have checked them and the output is the
same on both v17 and master. Do you think it would make sense to add
them as regression tests?
--
Regards,
Nazir Bilal Yavuz
Microsoft
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 22:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-10 11:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-21 20:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Neil Conway <neil.conway@gmail.com>
2026-01-31 16:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 13:51 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 18:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 21:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-06 22:19 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 22:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-11 13:27 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-11 22:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-13 11:45 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-13 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 13:38 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-18 21:26 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 21:54 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 04:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-19 09:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 22:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-20 00:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-20 09:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-20 18:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-23 09:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 04:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-24 13:57 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 17:48 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-25 14:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-02 19:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-04 15:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-09 18:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-10 12:35 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-10 17:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 11:36 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 18:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 18:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 19:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 19:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 20:42 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-12 10:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-12 17:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-13 11:57 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-13 13:34 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-13 14:05 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-13 15:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Greg Burd <greg@burd.me>
2026-03-13 16:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
@ 2026-03-13 16:16 ` Nathan Bossart <nathandbossart@gmail.com>
2026-03-13 17:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Greg Burd <greg@burd.me>
1 sibling, 1 reply; 178+ messages in thread
From: Nathan Bossart @ 2026-03-13 16:16 UTC (permalink / raw)
To: Nazir Bilal Yavuz <byavuz81@gmail.com>; +Cc: Greg Burd <greg@burd.me>; Manni Wood <manni.wood@enterprisedb.com>; KAZAR Ayoub <ma_kazar@esi.dz>; Neil Conway <neil.conway@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
On Fri, Mar 13, 2026 at 07:14:06PM +0300, Nazir Bilal Yavuz wrote:
> On Fri, 13 Mar 2026 at 18:29, Greg Burd <greg@burd.me> wrote:
>> I also created a few tests (attached) to check boundary conditions, I
>> might add some along with the RISC-V work.
>
> Thank you for the tests! I have checked them and the output is the
> same on both v17 and master. Do you think it would make sense to add
> them as regression tests?
Seems like a good idea. I was curious what the test coverage looked like
without extra tests. Once there's a report, we could choose a subset of
these to close any gaps.
--
nathan
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 22:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-10 11:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-21 20:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Neil Conway <neil.conway@gmail.com>
2026-01-31 16:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 13:51 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 18:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 21:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-06 22:19 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 22:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-11 13:27 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-11 22:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-13 11:45 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-13 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 13:38 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-18 21:26 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 21:54 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 04:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-19 09:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 22:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-20 00:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-20 09:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-20 18:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-23 09:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 04:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-24 13:57 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 17:48 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-25 14:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-02 19:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-04 15:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-09 18:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-10 12:35 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-10 17:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 11:36 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 18:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 18:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 19:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 19:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 20:42 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-12 10:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-12 17:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-13 11:57 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-13 13:34 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-13 14:05 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-13 15:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Greg Burd <greg@burd.me>
2026-03-13 16:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-13 16:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
@ 2026-03-13 17:22 ` Greg Burd <greg@burd.me>
0 siblings, 0 replies; 178+ messages in thread
From: Greg Burd @ 2026-03-13 17:22 UTC (permalink / raw)
To: Nathan Bossart <nathandbossart@gmail.com>; Nazir Bilal Yavuz <byavuz81@gmail.com>; +Cc: Manni Wood <manni.wood@enterprisedb.com>; KAZAR Ayoub <ma_kazar@esi.dz>; Neil Conway <neil.conway@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
On Fri, Mar 13, 2026, at 12:16 PM, Nathan Bossart wrote:
> On Fri, Mar 13, 2026 at 07:14:06PM +0300, Nazir Bilal Yavuz wrote:
>> On Fri, 13 Mar 2026 at 18:29, Greg Burd <greg@burd.me> wrote:
>>> I also created a few tests (attached) to check boundary conditions, I
>>> might add some along with the RISC-V work.
>>
>> Thank you for the tests! I have checked them and the output is the
>> same on both v17 and master. Do you think it would make sense to add
>> them as regression tests?
>
> Seems like a good idea. I was curious what the test coverage looked like
> without extra tests. Once there's a report, we could choose a subset of
> these to close any gaps.
+1, you said it better than I did! :)
> --
> nathan
best.
-greg
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 22:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-10 11:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-21 20:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Neil Conway <neil.conway@gmail.com>
2026-01-31 16:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 13:51 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 18:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 21:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-06 22:19 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 22:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-11 13:27 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-11 22:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-13 11:45 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-13 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 13:38 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-18 21:26 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 21:54 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 04:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-19 09:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 22:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-20 00:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-20 09:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-20 18:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-23 09:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 04:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-24 13:57 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 17:48 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-25 14:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-02 19:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-04 15:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-09 18:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-10 12:35 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-10 17:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 11:36 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 18:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 18:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 19:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 19:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 20:42 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-12 10:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-12 17:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-13 11:57 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-13 13:34 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-13 14:05 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-13 15:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Greg Burd <greg@burd.me>
2026-03-13 16:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
@ 2026-03-13 17:21 ` Greg Burd <greg@burd.me>
2026-03-13 17:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
1 sibling, 1 reply; 178+ messages in thread
From: Greg Burd @ 2026-03-13 17:21 UTC (permalink / raw)
To: Nazir Bilal Yavuz <byavuz81@gmail.com>; +Cc: Nathan Bossart <nathandbossart@gmail.com>; Manni Wood <manni.wood@enterprisedb.com>; KAZAR Ayoub <ma_kazar@esi.dz>; Neil Conway <neil.conway@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
On Fri, Mar 13, 2026, at 12:14 PM, Nazir Bilal Yavuz wrote:
> Hi Greg,
Hello Nazir,
> On Fri, 13 Mar 2026 at 18:29, Greg Burd <greg@burd.me> wrote:
>>
>> I've always been a fan of these kinds of optimization so I couldn't resist reviewing, but I know you're ready to commit so I'll just check on some systems I have. :)
>
> Thank you for the review!
Thank YOU for the work fixing this. :)
>> At first glance the implementation seems conservative, but correct and safe. Local testing on on Linux/FreeBSD x86_64, and Win11/aarch64/MSVC seem good. I also tried IllumOS/SPARCv9 and with some fixes (from another active thread) to the build system and it worked just fine too. I'm sure the 10 people care will be thrilled. ;-
>
> Yes, we can probably improve this further with heuristics, but for now
> we wanted to avoid introducing any potential regressions.
>> I also created a few tests (attached) to check boundary conditions, I might add some along with the RISC-V work.
>
> Thank you for the tests! I have checked them and the output is the
> same on both v17 and master. Do you think it would make sense to add
> them as regression tests?
If there are tests that materially add to the coverage that's a good thing to consider adding. I don't think all those tests are necessary.
best.
-greg
> --
> Regards,
> Nazir Bilal Yavuz
> Microsoft
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 22:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-10 11:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-21 20:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Neil Conway <neil.conway@gmail.com>
2026-01-31 16:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 13:51 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 18:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 21:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-06 22:19 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 22:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-11 13:27 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-11 22:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-13 11:45 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-13 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 13:38 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-18 21:26 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 21:54 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 04:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-19 09:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 22:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-20 00:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-20 09:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-20 18:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-23 09:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 04:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-24 13:57 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 17:48 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-25 14:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-02 19:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-04 15:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-09 18:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-10 12:35 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-10 17:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 11:36 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 18:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 18:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 19:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 19:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 20:42 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-12 10:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-12 17:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-13 11:57 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-13 13:34 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-13 14:05 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-13 15:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Greg Burd <greg@burd.me>
2026-03-13 16:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-13 17:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Greg Burd <greg@burd.me>
@ 2026-03-13 17:33 ` Nathan Bossart <nathandbossart@gmail.com>
2026-08-31 14:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
0 siblings, 1 reply; 178+ messages in thread
From: Nathan Bossart @ 2026-03-13 17:33 UTC (permalink / raw)
To: Greg Burd <greg@burd.me>; +Cc: Nazir Bilal Yavuz <byavuz81@gmail.com>; Manni Wood <manni.wood@enterprisedb.com>; KAZAR Ayoub <ma_kazar@esi.dz>; Neil Conway <neil.conway@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
On Fri, Mar 13, 2026 at 01:21:38PM -0400, Greg Burd wrote:
> On Fri, Mar 13, 2026, at 12:14 PM, Nazir Bilal Yavuz wrote:
>> On Fri, 13 Mar 2026 at 18:29, Greg Burd <greg@burd.me> wrote:
>>> I also created a few tests (attached) to check boundary conditions, I
>>> might add some along with the RISC-V work.
>>
>> Thank you for the tests! I have checked them and the output is the
>> same on both v17 and master. Do you think it would make sense to add
>> them as regression tests?
>
> If there are tests that materially add to the coverage that's a good
> thing to consider adding. I don't think all those tests are necessary.
We seem to have good coverage on the new code [0]. I still wouldn't mind
adding a couple of tests for correctness, if folks want them.
[0] https://coverage.postgresql.org/src/backend/commands/copyfromparse.c.gcov.html
--
nathan
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 22:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-10 11:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-21 20:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Neil Conway <neil.conway@gmail.com>
2026-01-31 16:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 13:51 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 18:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 21:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-06 22:19 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 22:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-11 13:27 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-11 22:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-13 11:45 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-13 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 13:38 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-18 21:26 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 21:54 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 04:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-19 09:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 22:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-20 00:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-20 09:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-20 18:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-23 09:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 04:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-24 13:57 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 17:48 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-25 14:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-02 19:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-04 15:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-09 18:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-10 12:35 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-10 17:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 11:36 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 18:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 18:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 19:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 19:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 20:42 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-12 10:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-12 17:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-13 11:57 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-13 13:34 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-13 14:05 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-13 15:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Greg Burd <greg@burd.me>
2026-03-13 16:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-13 17:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Greg Burd <greg@burd.me>
2026-03-13 17:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
@ 2026-08-31 14:24 ` Nathan Bossart <nathandbossart@gmail.com>
2026-08-31 17:48 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
0 siblings, 1 reply; 178+ messages in thread
From: Nathan Bossart @ 2026-08-31 14:24 UTC (permalink / raw)
To: Greg Burd <greg@burd.me>; +Cc: Nazir Bilal Yavuz <byavuz81@gmail.com>; Manni Wood <manni.wood@enterprisedb.com>; KAZAR Ayoub <ma_kazar@esi.dz>; Neil Conway <neil.conway@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
Claude found this bug:
CREATE TABLE t (a TEXT);
COPY t FROM PROGRAM $$printf 'abcdefghijklmnopqrst\n\\.\nxxxxxxxxxxxx\200'$$;
fails with
ERROR: invalid byte sequence for encoding "UTF8": 0x80
CONTEXT: COPY t, line 2
even though COPY should ignore everything after the \.
The best fix I could find involves teaching CopyLoadInputBuf() to avoid
erroring in the read-ahead path. AFAICT that doesn't meaningfully change
the performance characteristics. Patch attached.
--
nathan
From d9a9466a2f986c24d996110ffd691c5965509e8b Mon Sep 17 00:00:00 2001
From: Nathan Bossart <nathan@postgresql.org>
Date: Mon, 31 Aug 2026 09:08:23 -0500
Subject: [PATCH v1 1/1] Make COPY FROM's SIMD read-ahead speculative.
Commit e0a3a3fd53 added a SIMD scan for COPY FROM (FORMAT
{text,csv}) that refills the input buffer whenever fewer than
sizeof(Vector8) bytes remain, rather than when it has actually run
out. Presently, that read-ahead calls CopyLoadInputBuf() where
the scalar loop would not, and CopyLoadInputBuf() is where a
deferred encoding error is reported. So a file whose data ends
with \. followed by bytes that are invalid in the encoding, a case
CopyConvertBuf() goes out of its way to tolerate, now fails.
Whether it fails depends on how much never-examined padding sits
between the marker and the invalid byte: with 16-byte vectors, 12
bytes or fewer and the COPY errors out, 13 or more and it
succeeds.
To fix, teach CopyLoadInputBuf() to tell a speculative load, made
on the chance that the caller will want the data, from one the
caller needs in order to make progress. A speculative load leaves
a pending encoding error pending, so long as the caller still has
bytes to chew on. The SIMD path then hands those bytes to the
scalar code, which asks only for what it needs and so never reads
past the marker. Input that is really read is unaffected, except
that its errors are once again reported against the line holding
the bad byte, as they were before e0a3a3fd53.
The obvious fix, declining to read ahead at all, is worse than it
looks. The SIMD helper runs once per line, so refusing to refill
hands the entire remainder of any line that straddles a buffer
boundary to the scalar loop, and COPY of megabyte-wide lines
slowed by about 80% in my testing. Consuming the sub-vector tail
inside the helper instead cost the compiler its unrolling of the
vector loop, which was worse again. Leaving that loop untouched,
as this patch does, measures within noise of unpatched on every
line width I tried.
Oversight in commit e0a3a3fd53.
Discussion: https://postgr.es/m/CAOzEurSW8cNr6TPKsjrstnPfhf4QyQqB4tnPXGGe8N4e_v7Jig%40mail.gmail.com
Backpatch-through: 19
---
src/backend/commands/copyfromparse.c | 18 ++++++++++++++----
1 file changed, 14 insertions(+), 4 deletions(-)
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index 37750cca13a..dc02838bc38 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -167,7 +167,7 @@ static int CopyGetData(CopyFromState cstate, void *databuf,
int minread, int maxread);
static inline bool CopyGetInt32(CopyFromState cstate, int32 *val);
static inline bool CopyGetInt16(CopyFromState cstate, int16 *val);
-static void CopyLoadInputBuf(CopyFromState cstate);
+static void CopyLoadInputBuf(CopyFromState cstate, bool speculative);
static int CopyReadBinaryData(CopyFromState cstate, char *dest, int nbytes);
void
@@ -651,9 +651,15 @@ CopyLoadRawBuf(CopyFromState cstate)
*
* If INPUT_BUF_BYTES(cstate) > 0, the unprocessed bytes are moved to the start
* of the buffer and then we load more data after that.
+ *
+ * A speculative load is one made on the chance that the caller will want the
+ * data, not because it needs it yet. Such a load leaves a pending encoding
+ * error pending, so long as the caller still has something to chew on, since
+ * the input may never be read that far. NB: a speculative caller must cope
+ * with getting no additional data.
*/
static void
-CopyLoadInputBuf(CopyFromState cstate)
+CopyLoadInputBuf(CopyFromState cstate, bool speculative)
{
int nbytes = INPUT_BUF_BYTES(cstate);
@@ -684,7 +690,11 @@ CopyLoadInputBuf(CopyFromState cstate)
* conversion error.
*/
if (cstate->input_reached_error)
+ {
+ if (speculative && INPUT_BUF_BYTES(cstate) > 0)
+ return;
CopyConversionError(cstate);
+ }
/* no more input, and everything has been converted */
if (cstate->input_reached_eof)
@@ -1381,7 +1391,7 @@ CopyReadLineTextSIMDHelper(CopyFromState cstate, bool is_csv,
{
REFILL_LINEBUF;
- CopyLoadInputBuf(cstate);
+ CopyLoadInputBuf(cstate, true);
/* update our local variables */
*hit_eof_p = cstate->input_reached_eof;
input_buf_ptr = cstate->input_buf_index;
@@ -1566,7 +1576,7 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
{
REFILL_LINEBUF;
- CopyLoadInputBuf(cstate);
+ CopyLoadInputBuf(cstate, false);
/* update our local variables */
hit_eof = cstate->input_reached_eof;
input_buf_ptr = cstate->input_buf_index;
--
2.55.0
Attachments:
[text/plain] v1-0001-Make-COPY-FROM-s-SIMD-read-ahead-speculative.patch (4.5K, ../../apWOr1kgVSqweBnU@nathan/2-v1-0001-Make-COPY-FROM-s-SIMD-read-ahead-speculative.patch)
download | inline diff:
From d9a9466a2f986c24d996110ffd691c5965509e8b Mon Sep 17 00:00:00 2001
From: Nathan Bossart <nathan@postgresql.org>
Date: Mon, 31 Aug 2026 09:08:23 -0500
Subject: [PATCH v1 1/1] Make COPY FROM's SIMD read-ahead speculative.
Commit e0a3a3fd53 added a SIMD scan for COPY FROM (FORMAT
{text,csv}) that refills the input buffer whenever fewer than
sizeof(Vector8) bytes remain, rather than when it has actually run
out. Presently, that read-ahead calls CopyLoadInputBuf() where
the scalar loop would not, and CopyLoadInputBuf() is where a
deferred encoding error is reported. So a file whose data ends
with \. followed by bytes that are invalid in the encoding, a case
CopyConvertBuf() goes out of its way to tolerate, now fails.
Whether it fails depends on how much never-examined padding sits
between the marker and the invalid byte: with 16-byte vectors, 12
bytes or fewer and the COPY errors out, 13 or more and it
succeeds.
To fix, teach CopyLoadInputBuf() to tell a speculative load, made
on the chance that the caller will want the data, from one the
caller needs in order to make progress. A speculative load leaves
a pending encoding error pending, so long as the caller still has
bytes to chew on. The SIMD path then hands those bytes to the
scalar code, which asks only for what it needs and so never reads
past the marker. Input that is really read is unaffected, except
that its errors are once again reported against the line holding
the bad byte, as they were before e0a3a3fd53.
The obvious fix, declining to read ahead at all, is worse than it
looks. The SIMD helper runs once per line, so refusing to refill
hands the entire remainder of any line that straddles a buffer
boundary to the scalar loop, and COPY of megabyte-wide lines
slowed by about 80% in my testing. Consuming the sub-vector tail
inside the helper instead cost the compiler its unrolling of the
vector loop, which was worse again. Leaving that loop untouched,
as this patch does, measures within noise of unpatched on every
line width I tried.
Oversight in commit e0a3a3fd53.
Discussion: https://postgr.es/m/CAOzEurSW8cNr6TPKsjrstnPfhf4QyQqB4tnPXGGe8N4e_v7Jig%40mail.gmail.com
Backpatch-through: 19
---
src/backend/commands/copyfromparse.c | 18 ++++++++++++++----
1 file changed, 14 insertions(+), 4 deletions(-)
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index 37750cca13a..dc02838bc38 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -167,7 +167,7 @@ static int CopyGetData(CopyFromState cstate, void *databuf,
int minread, int maxread);
static inline bool CopyGetInt32(CopyFromState cstate, int32 *val);
static inline bool CopyGetInt16(CopyFromState cstate, int16 *val);
-static void CopyLoadInputBuf(CopyFromState cstate);
+static void CopyLoadInputBuf(CopyFromState cstate, bool speculative);
static int CopyReadBinaryData(CopyFromState cstate, char *dest, int nbytes);
void
@@ -651,9 +651,15 @@ CopyLoadRawBuf(CopyFromState cstate)
*
* If INPUT_BUF_BYTES(cstate) > 0, the unprocessed bytes are moved to the start
* of the buffer and then we load more data after that.
+ *
+ * A speculative load is one made on the chance that the caller will want the
+ * data, not because it needs it yet. Such a load leaves a pending encoding
+ * error pending, so long as the caller still has something to chew on, since
+ * the input may never be read that far. NB: a speculative caller must cope
+ * with getting no additional data.
*/
static void
-CopyLoadInputBuf(CopyFromState cstate)
+CopyLoadInputBuf(CopyFromState cstate, bool speculative)
{
int nbytes = INPUT_BUF_BYTES(cstate);
@@ -684,7 +690,11 @@ CopyLoadInputBuf(CopyFromState cstate)
* conversion error.
*/
if (cstate->input_reached_error)
+ {
+ if (speculative && INPUT_BUF_BYTES(cstate) > 0)
+ return;
CopyConversionError(cstate);
+ }
/* no more input, and everything has been converted */
if (cstate->input_reached_eof)
@@ -1381,7 +1391,7 @@ CopyReadLineTextSIMDHelper(CopyFromState cstate, bool is_csv,
{
REFILL_LINEBUF;
- CopyLoadInputBuf(cstate);
+ CopyLoadInputBuf(cstate, true);
/* update our local variables */
*hit_eof_p = cstate->input_reached_eof;
input_buf_ptr = cstate->input_buf_index;
@@ -1566,7 +1576,7 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
{
REFILL_LINEBUF;
- CopyLoadInputBuf(cstate);
+ CopyLoadInputBuf(cstate, false);
/* update our local variables */
hit_eof = cstate->input_reached_eof;
input_buf_ptr = cstate->input_buf_index;
--
2.55.0
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 22:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-10 11:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-21 20:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Neil Conway <neil.conway@gmail.com>
2026-01-31 16:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 13:51 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 18:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 21:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-06 22:19 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 22:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-11 13:27 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-11 22:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-13 11:45 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-13 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 13:38 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-18 21:26 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 21:54 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 04:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-19 09:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 22:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-20 00:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-20 09:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-20 18:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-23 09:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 04:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-24 13:57 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 17:48 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-25 14:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-02 19:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-04 15:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-09 18:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-10 12:35 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-10 17:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 11:36 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 18:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 18:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 19:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 19:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 20:42 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-12 10:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-12 17:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-13 11:57 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-13 13:34 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-13 14:05 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-13 15:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Greg Burd <greg@burd.me>
2026-03-13 16:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-13 17:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Greg Burd <greg@burd.me>
2026-03-13 17:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-08-31 14:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
@ 2026-08-31 17:48 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-09-01 04:41 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Greg Burd <greg@burd.me>
0 siblings, 1 reply; 178+ messages in thread
From: Nazir Bilal Yavuz @ 2026-08-31 17:48 UTC (permalink / raw)
To: Nathan Bossart <nathandbossart@gmail.com>; +Cc: Greg Burd <greg@burd.me>; Manni Wood <manni.wood@enterprisedb.com>; KAZAR Ayoub <ma_kazar@esi.dz>; Neil Conway <neil.conway@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
Hi,
On Mon, 31 Aug 2026 at 17:24, Nathan Bossart <nathandbossart@gmail.com> wrote:
>
> Claude found this bug:
>
> CREATE TABLE t (a TEXT);
> COPY t FROM PROGRAM $$printf 'abcdefghijklmnopqrst\n\\.\nxxxxxxxxxxxx\200'$$;
>
> fails with
>
> ERROR: invalid byte sequence for encoding "UTF8": 0x80
> CONTEXT: COPY t, line 2
>
> even though COPY should ignore everything after the \.
Nice catch! I can reproduce the problem and confirm that the patch fixes it.
> The best fix I could find involves teaching CopyLoadInputBuf() to avoid
> erroring in the read-ahead path. AFAICT that doesn't meaningfully change
> the performance characteristics. Patch attached.
That seems like the easiest fix without complicating the code. I will
review this code in more detail tomorrow, but I think adding the
reproducer above as a test case makes sense to me.
--
Regards,
Nazir Bilal Yavuz
Microsoft
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 22:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-10 11:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-21 20:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Neil Conway <neil.conway@gmail.com>
2026-01-31 16:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 13:51 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 18:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 21:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-06 22:19 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 22:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-11 13:27 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-11 22:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-13 11:45 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-13 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 13:38 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-18 21:26 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 21:54 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 04:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-19 09:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 22:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-20 00:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-20 09:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-20 18:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-23 09:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 04:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-24 13:57 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 17:48 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-25 14:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-02 19:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-04 15:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-09 18:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-10 12:35 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-10 17:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 11:36 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 18:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 18:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 19:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 19:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 20:42 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-12 10:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-12 17:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-13 11:57 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-13 13:34 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-13 14:05 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-13 15:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Greg Burd <greg@burd.me>
2026-03-13 16:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-13 17:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Greg Burd <greg@burd.me>
2026-03-13 17:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-08-31 14:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-08-31 17:48 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
@ 2026-09-01 04:41 ` Greg Burd <greg@burd.me>
2026-09-01 08:17 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
0 siblings, 1 reply; 178+ messages in thread
From: Greg Burd @ 2026-09-01 04:41 UTC (permalink / raw)
To: Nazir Bilal Yavuz <byavuz81@gmail.com>; Nathan Bossart <nathandbossart@gmail.com>; +Cc: Manni Wood <manni.wood@enterprisedb.com>; KAZAR Ayoub <ma_kazar@esi.dz>; Neil Conway <neil.conway@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
Hey Nathan, Nazir,
I've also reproduced the error and confirmed the patch addresses it
cleanly.
For anyone following along, the mechanism: commit e0a3a3fd53 added
CopyReadLineTextSIMDHelper, which refills the input buffer once fewer
than sizeof(Vector8) bytes remain:
if (copy_buf_len - input_buf_ptr < sizeof(Vector8)) /* reads ahead */
whereas scalar CopyReadLineText only refills when actually empty:
if (input_buf_ptr >= copy_buf_len || need_data)
That difference is the issue. CopyConvertBuf() deliberately tolerates an
invalid byte sequence after the \. end-of-copy marker (see the comment
near its convertedlen == 0 block), deferring the error. But
CopyLoadInputBuf() raises it the moment input_reached_error is set. The
SIMD read-ahead reaches that raise before the scalar loop has consumed
\., so a file ending in \.\n<valid bytes><invalid byte> errors out where
the scalar path would have stopped cleanly at \.
+1 on adding the reproducer as a test and calling it good.
best,
-greg
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 22:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-10 11:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-21 20:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Neil Conway <neil.conway@gmail.com>
2026-01-31 16:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 13:51 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 18:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 21:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-06 22:19 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 22:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-11 13:27 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-11 22:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-13 11:45 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-13 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 13:38 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-18 21:26 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 21:54 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 04:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-19 09:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 22:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-20 00:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-20 09:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-20 18:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-23 09:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 04:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-24 13:57 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 17:48 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-25 14:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-02 19:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-04 15:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-09 18:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-10 12:35 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-10 17:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 11:36 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 18:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 18:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 19:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 19:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 20:42 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-12 10:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-12 17:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-13 11:57 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-13 13:34 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-13 14:05 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-13 15:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Greg Burd <greg@burd.me>
2026-03-13 16:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-13 17:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Greg Burd <greg@burd.me>
2026-03-13 17:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-08-31 14:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-08-31 17:48 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-09-01 04:41 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Greg Burd <greg@burd.me>
@ 2026-09-01 08:17 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-09-01 19:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
0 siblings, 1 reply; 178+ messages in thread
From: Nazir Bilal Yavuz @ 2026-09-01 08:17 UTC (permalink / raw)
To: Greg Burd <greg@burd.me>; +Cc: Nathan Bossart <nathandbossart@gmail.com>; Manni Wood <manni.wood@enterprisedb.com>; KAZAR Ayoub <ma_kazar@esi.dz>; Neil Conway <neil.conway@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
Hi,
On Tue, 1 Sept 2026 at 07:41, Greg Burd <greg@burd.me> wrote:
>
> For anyone following along, the mechanism: commit e0a3a3fd53 added
> CopyReadLineTextSIMDHelper, which refills the input buffer once fewer
> than sizeof(Vector8) bytes remain:
>
> if (copy_buf_len - input_buf_ptr < sizeof(Vector8)) /* reads ahead */
>
> whereas scalar CopyReadLineText only refills when actually empty:
>
> if (input_buf_ptr >= copy_buf_len || need_data)
>
> That difference is the issue. CopyConvertBuf() deliberately tolerates an
> invalid byte sequence after the \. end-of-copy marker (see the comment
> near its convertedlen == 0 block), deferring the error. But
> CopyLoadInputBuf() raises it the moment input_reached_error is set. The
> SIMD read-ahead reaches that raise before the scalar loop has consumed
> \., so a file ending in \.\n<valid bytes><invalid byte> errors out where
> the scalar path would have stopped cleanly at \.
Thank you for great explanation!
On Mon, 31 Aug 2026 at 20:48, Nazir Bilal Yavuz <byavuz81@gmail.com> wrote:
>
> That seems like the easiest fix without complicating the code. I will
> review this code in more detail tomorrow, but I think adding the
> reproducer above as a test case makes sense to me.
I re-checked the patch and the fix LGTM. My only comment is adding the
reproducer as a test case.
--
Regards,
Nazir Bilal Yavuz
Microsoft
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 22:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-10 11:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-21 20:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Neil Conway <neil.conway@gmail.com>
2026-01-31 16:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 13:51 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 18:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 21:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-06 22:19 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 22:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-11 13:27 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-11 22:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-13 11:45 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-13 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 13:38 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-18 21:26 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 21:54 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 04:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-19 09:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 22:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-20 00:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-20 09:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-20 18:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-23 09:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 04:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-24 13:57 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 17:48 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-25 14:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-02 19:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-04 15:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-09 18:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-10 12:35 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-10 17:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 11:36 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 18:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 18:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 19:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 19:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 20:42 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-12 10:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-12 17:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-13 11:57 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-13 13:34 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-13 14:05 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-13 15:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Greg Burd <greg@burd.me>
2026-03-13 16:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-13 17:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Greg Burd <greg@burd.me>
2026-03-13 17:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-08-31 14:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-08-31 17:48 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-09-01 04:41 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Greg Burd <greg@burd.me>
2026-09-01 08:17 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
@ 2026-09-01 19:23 ` Nathan Bossart <nathandbossart@gmail.com>
2026-09-07 16:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
0 siblings, 1 reply; 178+ messages in thread
From: Nathan Bossart @ 2026-09-01 19:23 UTC (permalink / raw)
To: Nazir Bilal Yavuz <byavuz81@gmail.com>; +Cc: Greg Burd <greg@burd.me>; Manni Wood <manni.wood@enterprisedb.com>; KAZAR Ayoub <ma_kazar@esi.dz>; Neil Conway <neil.conway@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
On Tue, Sep 01, 2026 at 11:17:03AM +0300, Nazir Bilal Yavuz wrote:
> I re-checked the patch and the fix LGTM. My only comment is adding the
> reproducer as a test case.
Committed, thanks for looking. I'm concerned about portability and
encoding issues with such a test, so I didn't add it yet. But I'm not
opposed to adding a test along these lines as a separate effort.
--
nathan
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 22:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-10 11:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-21 20:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Neil Conway <neil.conway@gmail.com>
2026-01-31 16:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 13:51 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 18:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 21:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-06 22:19 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 22:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-11 13:27 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-11 22:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-13 11:45 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-13 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 13:38 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-18 21:26 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 21:54 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 04:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-19 09:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 22:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-20 00:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-20 09:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-20 18:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-23 09:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 04:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-24 13:57 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 17:48 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-25 14:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-02 19:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-04 15:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-09 18:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-10 12:35 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-10 17:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 11:36 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 18:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 18:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 19:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 19:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 20:42 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-12 10:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-12 17:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-13 11:57 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-13 13:34 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-13 14:05 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-13 15:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Greg Burd <greg@burd.me>
2026-03-13 16:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-13 17:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Greg Burd <greg@burd.me>
2026-03-13 17:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-08-31 14:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-08-31 17:48 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-09-01 04:41 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Greg Burd <greg@burd.me>
2026-09-01 08:17 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-09-01 19:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
@ 2026-09-07 16:29 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-09-08 10:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
0 siblings, 1 reply; 178+ messages in thread
From: Nazir Bilal Yavuz @ 2026-09-07 16:29 UTC (permalink / raw)
To: Nathan Bossart <nathandbossart@gmail.com>; +Cc: Greg Burd <greg@burd.me>; Manni Wood <manni.wood@enterprisedb.com>; KAZAR Ayoub <ma_kazar@esi.dz>; Neil Conway <neil.conway@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
Hi,
On Tue, 1 Sept 2026 at 22:24, Nathan Bossart <nathandbossart@gmail.com> wrote:
>
> On Tue, Sep 01, 2026 at 11:17:03AM +0300, Nazir Bilal Yavuz wrote:
> > I re-checked the patch and the fix LGTM. My only comment is adding the
> > reproducer as a test case.
>
> Committed, thanks for looking. I'm concerned about portability and
> encoding issues with such a test, so I didn't add it yet. But I'm not
> opposed to adding a test along these lines as a separate effort.
After the problem you found, I had another LLM review the code and it
found one issue:
When fewer than one SIMD vector of bytes (16 in this case) remain
buffered, CopyReadLineTextSIMDHelper() refills the input before
falling back to scalar parsing. If those buffered bytes already
contain a complete '\.' (end-of-copy marker) and the source is an open
FIFO with no more data, the refill blocks, so COPY fails to recognize
the marker until more data arrives or the writer closes the FIFO.
Reproducer SQL script is attached, you can see that this causes a
hang.
One potential fix is checking for an end-of-copy marker when fewer
than one SIMD vector of bytes remains in the buffer and we decide to
load more data into it. I haven't benchmarked this solution yet but it
could potentially cause a slowdown when we load data in smaller
chunks. Otherwise, I don't think this solution will cause a slowdown.
I am planning to work on this tomorrow.
Any opinions on the bug or the potential solution?
--
Regards,
Nazir Bilal Yavuz
Microsoft
Attachments:
[application/sql] reproducer.sql (772B, ../../CAN55FZ1qFb4Yo3-MWeQfGQnu1_Ksx4xcFC2m3hyw8a--+eHQYQ@mail.gmail.com/2-reproducer.sql)
download
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 22:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-10 11:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-21 20:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Neil Conway <neil.conway@gmail.com>
2026-01-31 16:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 13:51 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 18:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 21:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-06 22:19 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 22:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-11 13:27 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-11 22:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-13 11:45 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-13 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 13:38 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-18 21:26 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 21:54 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 04:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-19 09:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 22:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-20 00:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-20 09:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-20 18:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-23 09:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 04:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-24 13:57 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 17:48 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-25 14:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-02 19:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-04 15:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-09 18:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-10 12:35 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-10 17:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 11:36 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 18:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 18:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 19:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 19:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 20:42 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-12 10:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-12 17:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-13 11:57 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-13 13:34 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-13 14:05 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-13 15:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Greg Burd <greg@burd.me>
2026-03-13 16:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-13 17:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Greg Burd <greg@burd.me>
2026-03-13 17:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-08-31 14:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-08-31 17:48 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-09-01 04:41 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Greg Burd <greg@burd.me>
2026-09-01 08:17 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-09-01 19:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-09-07 16:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
@ 2026-09-08 10:11 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-09-08 15:08 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Greg Burd <greg@burd.me>
2026-09-08 15:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
0 siblings, 2 replies; 178+ messages in thread
From: Nazir Bilal Yavuz @ 2026-09-08 10:11 UTC (permalink / raw)
To: Nathan Bossart <nathandbossart@gmail.com>; +Cc: Greg Burd <greg@burd.me>; KAZAR Ayoub <ma_kazar@esi.dz>; Neil Conway <neil.conway@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
Hi,
On Mon, 7 Sept 2026 at 19:29, Nazir Bilal Yavuz <byavuz81@gmail.com> wrote:
>
> When fewer than one SIMD vector of bytes (16 in this case) remain
> buffered, CopyReadLineTextSIMDHelper() refills the input before
> falling back to scalar parsing. If those buffered bytes already
> contain a complete '\.' (end-of-copy marker) and the source is an open
> FIFO with no more data, the refill blocks, so COPY fails to recognize
> the marker until more data arrives or the writer closes the FIFO.
> Reproducer SQL script is attached, you can see that this causes a
> hang.
>
> One potential fix is checking for an end-of-copy marker when fewer
> than one SIMD vector of bytes remains in the buffer and we decide to
> load more data into it. I haven't benchmarked this solution yet but it
> could potentially cause a slowdown when we load data in smaller
> chunks. Otherwise, I don't think this solution will cause a slowdown.
> I am planning to work on this tomorrow.
>
> Any opinions on the bug or the potential solution?
Here is an attempt to solve this problem. I ran Manni's script and saw
a 1-2% slowdown on TEXT-wide inputs. It is still faster compared to
the version without the SIMD patch; the slowdown is relative to the
current master branch (which includes the SIMD patch).
--
Regards,
Nazir Bilal Yavuz
Microsoft
Attachments:
[text/x-patch] v1-0001-Fix-COPY-SIMD-refill-blocking-on-buffered-end-mar.patch (1.6K, ../../CAN55FZ3U2waD=5ZuVmju-HtZ4cO4onxHN4jXskk8Y11M6o3NOQ@mail.gmail.com/2-v1-0001-Fix-COPY-SIMD-refill-blocking-on-buffered-end-mar.patch)
download | inline diff:
From cef5dc405842841ca4355f9b0607e0919c349841 Mon Sep 17 00:00:00 2001
From: Nazir Bilal Yavuz <byavuz81@gmail.com>
Date: Tue, 8 Sep 2026 13:06:19 +0300
Subject: [PATCH v1] Fix COPY SIMD refill blocking on buffered end markers
Check short text tails for a backslash before refilling, leaving marker
recognition to the scalar parser. This avoids blocking on an open FIFO
with a buffered end marker while preserving SIMD refills for long lines.
Fixes a regression introduced by e0a3a3fd536.
---
src/backend/commands/copyfromparse.c | 13 +++++++++++++
1 file changed, 13 insertions(+)
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index 98bf30ef2e7..2001b6ce822 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -1390,6 +1390,19 @@ CopyReadLineTextSIMDHelper(CopyFromState cstate, bool is_csv,
/* Load more data if needed. */
if (copy_buf_len - input_buf_ptr < sizeof(Vector8))
{
+ /*
+ * In text mode, a backslash in the remaining bytes might start an
+ * end-of-copy marker. Check only for the backslash, leaving
+ * marker recognition and escape handling to the scalar code. Do
+ * this before reading more data, which could block on a pipe
+ * despite a complete marker being buffered. Otherwise, refill
+ * here so that long lines can continue to use SIMD.
+ */
+ if (!is_csv &&
+ memchr(copy_input_buf + input_buf_ptr, '\\',
+ copy_buf_len - input_buf_ptr) != NULL)
+ break;
+
REFILL_LINEBUF;
CopyLoadInputBuf(cstate, true);
--
2.47.3
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 22:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-10 11:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-21 20:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Neil Conway <neil.conway@gmail.com>
2026-01-31 16:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 13:51 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 18:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 21:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-06 22:19 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 22:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-11 13:27 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-11 22:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-13 11:45 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-13 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 13:38 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-18 21:26 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 21:54 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 04:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-19 09:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 22:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-20 00:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-20 09:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-20 18:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-23 09:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 04:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-24 13:57 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 17:48 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-25 14:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-02 19:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-04 15:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-09 18:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-10 12:35 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-10 17:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 11:36 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 18:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 18:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 19:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 19:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 20:42 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-12 10:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-12 17:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-13 11:57 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-13 13:34 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-13 14:05 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-13 15:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Greg Burd <greg@burd.me>
2026-03-13 16:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-13 17:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Greg Burd <greg@burd.me>
2026-03-13 17:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-08-31 14:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-08-31 17:48 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-09-01 04:41 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Greg Burd <greg@burd.me>
2026-09-01 08:17 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-09-01 19:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-09-07 16:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-09-08 10:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
@ 2026-09-08 15:08 ` Greg Burd <greg@burd.me>
2026-09-08 15:36 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
1 sibling, 1 reply; 178+ messages in thread
From: Greg Burd @ 2026-09-08 15:08 UTC (permalink / raw)
To: Nazir Bilal Yavuz <byavuz81@gmail.com>; Nathan Bossart <nathandbossart@gmail.com>; +Cc: KAZAR Ayoub <ma_kazar@esi.dz>; Neil Conway <neil.conway@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
Hey Nazir, Nathan,
Thanks for turning this around so fast. The v1 looks right to me.
Checking only for the backslash and letting the scalar parser do the
actual marker recognition is the right call. Trying to detect a full
'\.' in the SIMD helper would mean duplicating the CRLF handling and the
"marker is not alone on its line" checks, and that logic is fiddly enough
that I'd rather it live in exactly one place. The !is_csv guard is also
correctly scoped: the scalar path only treats '\' / '\.' specially in
text mode (copyfromparse.c, "if (c == '\\' && !is_csv)"), so CSV never
had a buffered marker to worry about and doesn't need the check.
On the perf side, I read your 1-2% on TEXT-wide as the expected cost:
the tail check bails to scalar on any backslash in the last < sizeof(Vector8)
bytes, not just on a real marker, so escaped data near a buffer boundary
gives up SIMD for that refill. It re-enables on the next line, so it's
confined to boundaries, and as you say it's still well ahead of pre-SIMD.
I think that's the right trade; correctness wins here. Might be worth a
sentence in the commit message noting the small text-path cost so it's on
the record.
One thing I'd like to see land with the fix: a test. I'll see if I can
knock one out if you don't first.
best,
-greg
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 22:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-10 11:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-21 20:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Neil Conway <neil.conway@gmail.com>
2026-01-31 16:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 13:51 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 18:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 21:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-06 22:19 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 22:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-11 13:27 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-11 22:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-13 11:45 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-13 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 13:38 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-18 21:26 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 21:54 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 04:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-19 09:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 22:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-20 00:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-20 09:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-20 18:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-23 09:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 04:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-24 13:57 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 17:48 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-25 14:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-02 19:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-04 15:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-09 18:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-10 12:35 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-10 17:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 11:36 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 18:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 18:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 19:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 19:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 20:42 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-12 10:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-12 17:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-13 11:57 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-13 13:34 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-13 14:05 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-13 15:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Greg Burd <greg@burd.me>
2026-03-13 16:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-13 17:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Greg Burd <greg@burd.me>
2026-03-13 17:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-08-31 14:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-08-31 17:48 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-09-01 04:41 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Greg Burd <greg@burd.me>
2026-09-01 08:17 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-09-01 19:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-09-07 16:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-09-08 10:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-09-08 15:08 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Greg Burd <greg@burd.me>
@ 2026-09-08 15:36 ` Nathan Bossart <nathandbossart@gmail.com>
2026-09-08 15:58 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Greg Burd <greg@burd.me>
0 siblings, 1 reply; 178+ messages in thread
From: Nathan Bossart @ 2026-09-08 15:36 UTC (permalink / raw)
To: Greg Burd <greg@burd.me>; +Cc: Nazir Bilal Yavuz <byavuz81@gmail.com>; KAZAR Ayoub <ma_kazar@esi.dz>; Neil Conway <neil.conway@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
On Tue, Sep 08, 2026 at 11:08:36AM -0400, Greg Burd wrote:
> One thing I'd like to see land with the fix: a test. I'll see if I can
> knock one out if you don't first.
I'm open to adding a test, but I'm not going to hold up the patch for it.
I've been reluctant to add too many tests for corner cases here because I'm
concerned about portability and fragility. But I will admit that I haven't
tried too hard...
--
nathan
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 22:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-10 11:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-21 20:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Neil Conway <neil.conway@gmail.com>
2026-01-31 16:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 13:51 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 18:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 21:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-06 22:19 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 22:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-11 13:27 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-11 22:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-13 11:45 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-13 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 13:38 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-18 21:26 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 21:54 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 04:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-19 09:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 22:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-20 00:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-20 09:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-20 18:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-23 09:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 04:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-24 13:57 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 17:48 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-25 14:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-02 19:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-04 15:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-09 18:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-10 12:35 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-10 17:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 11:36 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 18:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 18:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 19:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 19:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 20:42 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-12 10:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-12 17:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-13 11:57 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-13 13:34 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-13 14:05 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-13 15:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Greg Burd <greg@burd.me>
2026-03-13 16:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-13 17:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Greg Burd <greg@burd.me>
2026-03-13 17:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-08-31 14:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-08-31 17:48 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-09-01 04:41 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Greg Burd <greg@burd.me>
2026-09-01 08:17 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-09-01 19:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-09-07 16:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-09-08 10:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-09-08 15:08 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Greg Burd <greg@burd.me>
2026-09-08 15:36 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
@ 2026-09-08 15:58 ` Greg Burd <greg@burd.me>
2026-09-08 20:58 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
0 siblings, 1 reply; 178+ messages in thread
From: Greg Burd @ 2026-09-08 15:58 UTC (permalink / raw)
To: Nathan Bossart <nathandbossart@gmail.com>; +Cc: Nazir Bilal Yavuz <byavuz81@gmail.com>; KAZAR Ayoub <ma_kazar@esi.dz>; Neil Conway <neil.conway@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
On Tue, Sep 8, 2026, at 11:36 AM, Nathan Bossart wrote:
> On Tue, Sep 08, 2026 at 11:08:36AM -0400, Greg Burd wrote:
>> One thing I'd like to see land with the fix: a test. I'll see if I can
>> knock one out if you don't first.
>
> I'm open to adding a test, but I'm not going to hold up the patch for it.
> I've been reluctant to add too many tests for corner cases here because I'm
> concerned about portability and fragility. But I will admit that I haven't
> tried too hard...
Hey Nathan, I like your approach better.
The test writes one full input buffer (INPUT_BUF_SIZE, 65536 bytes) down a
COPY FROM PROGRAM pipe: a 65524-byte line, then a complete \. marker, then
8 trailing bytes. That leaves 11 bytes unexamined when the SIMD path wants
to refill, which is under sizeof(Vector8) and already holds the whole
marker. The writer then dribbles a byte every 50ms and keeps the pipe open,
so a speculative load blocks rather than seeing EOF. statement_timeout
bounds the failure, so a regression fails the test instead of hanging the
suite. Skipped on Windows, where the blocking pipe and SIGPIPE behavior
isn't reliable.
I confirmed both directions, with the fix reverse-applied it fails with
"canceling statement due to statement timeout" at COPY line 2, and with the
fix in place it passes in under 2s. I also instrumented the early return in
CopyLoadInputBuf() to be sure the test actually reaches it, and it fires
once with nbytes=11.
Attached is a TAP test and your patch as v3.
-greg
> --
> nathan
Attachments:
[application/octet-stream] v3-0001-Fix-hangs-in-COPY-FROM-FORMAT-text.patch (3.3K, ../../633c2a60-e5b9-49d2-a93d-946d184d946d@app.fastmail.com/2-v3-0001-Fix-hangs-in-COPY-FROM-FORMAT-text.patch)
download | inline diff:
From e8ec3f600493a38dcaa02625b302ca6214848edf Mon Sep 17 00:00:00 2001
From: Nathan Bossart <nathan@postgresql.org>
Date: Tue, 8 Sep 2026 10:24:31 -0500
Subject: [PATCH v3 1/2] Fix hangs in COPY FROM (FORMAT text).
The SIMD path for this command reads ahead via CopyLoadInputBuf()
whenever fewer than sizeof(Vector8) bytes remain in the input
buffer, even if those bytes hold a complete end-of-copy marker.
If the input is a pipe whose writer has sent the marker but not
closed the pipe, that read blocks, and COPY waits for data it will
never use. The scalar loop asks only for the bytes it needs, so
it stops at the marker without reading any further.
To fix, teach CopyLoadInputBuf() to decline a speculative load
when the caller's remaining bytes contain a backslash, which in
text mode might begin such a marker. (CSV mode doesn't treat
\. as special, so it is unaffected.) The SIMD path then hands
those bytes to the scalar loop, as it already does when a load
comes up short. Checking for the marker in the SIMD helper itself
would be more direct, but a call there costs the compiler
registers on every line, which measurably slowed COPY of short
lines in my testing.
Oversight in commit e0a3a3fd53.
Author: Nazir Bilal Yavuz <byavuz81@gmail.com>
Discussion: https://postgr.es/m/CAN55FZ1qFb4Yo3-MWeQfGQnu1_Ksx4xcFC2m3hyw8a--%2BeHQYQ%40mail.gmail.com
Backpatch-through: 19
---
src/backend/commands/copyfromparse.c | 22 ++++++++++++++++++----
1 file changed, 18 insertions(+), 4 deletions(-)
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index 98bf30ef2e7..68088108c7c 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -652,16 +652,30 @@ CopyLoadRawBuf(CopyFromState cstate)
* If INPUT_BUF_BYTES(cstate) > 0, the unprocessed bytes are moved to the start
* of the buffer and then we load more data after that.
*
- * If "speculative" is true, this function skips reporting any encoding or
- * conversion errors, provided there are still data for the caller to process.
- * Such callers must be prepared for this function to return without loading
- * anything.
+ * If "speculative" is true, the caller has not yet examined the data left in
+ * input_buf and needn't load more to make progress. In that case, this
+ * function skips reporting any encoding or conversion errors, and in text
+ * mode it declines to read past a backslash, as that might begin an
+ * end-of-copy marker. Such callers must be prepared for this function to
+ * return without loading anything.
*/
static void
CopyLoadInputBuf(CopyFromState cstate, bool speculative)
{
int nbytes = INPUT_BUF_BYTES(cstate);
+ /*
+ * In text mode, a backslash among the bytes a speculative caller has yet
+ * to examine might begin an end-of-copy marker. The caller must find
+ * that on its own, without waiting on input that may never arrive, as
+ * from a pipe whose writer has sent the marker but not closed the pipe.
+ * So in that case, just return without loading anything.
+ */
+ if (speculative && cstate->opts.format == COPY_FORMAT_TEXT &&
+ memchr(cstate->input_buf + cstate->input_buf_index, '\\',
+ nbytes) != NULL)
+ return;
+
/*
* The caller has updated input_buf_index to indicate how much of the
* input has been consumed and isn't needed anymore. If input_buf is the
--
2.50.1 (Apple Git-155)
[application/octet-stream] v3-0002-Add-TAP-test-for-COPY-SIMD-buffered-end-of-copy-m.patch (5.1K, ../../633c2a60-e5b9-49d2-a93d-946d184d946d@app.fastmail.com/3-v3-0002-Add-TAP-test-for-COPY-SIMD-buffered-end-of-copy-m.patch)
download | inline diff:
From 8b93b6a63fcf9ee3ec5a9095bc47bd203e9976ba Mon Sep 17 00:00:00 2001
From: Greg Burd <greg@burd.me>
Date: Tue, 8 Sep 2026 11:10:34 -0400
Subject: [PATCH v3 2/2] Add TAP test for COPY SIMD buffered end-of-copy marker
The earlier commit fixed a hang where CopyReadLineTextSIMDHelper()
refilled the input buffer past a complete "\." marker already sitting in
the short tail, blocking on a still-open pipe. No tests in the tree
cover this code.
The test writes one full input buffer with the marker in that tail, then
keeps the pipe open with a slow dribble so a bad refill blocks rather
than seeing EOF. statement_timeout bounds the failure, so a regression
fails instead of hanging the suite. Skipped on Windows, where the
blocking pipe and SIGPIPE behavior is unreliable.
Discussion: https://postgr.es/m/CAN55FZ3U2waD%3D5ZuVmju-HtZ4cO4onxHN4jXskk8Y11M6o3NOQ%40mail.gmail.com
---
src/test/modules/test_misc/meson.build | 1 +
.../test_misc/t/016_copy_simd_eoc_marker.pl | 90 +++++++++++++++++++
2 files changed, 91 insertions(+)
create mode 100644 src/test/modules/test_misc/t/016_copy_simd_eoc_marker.pl
diff --git a/src/test/modules/test_misc/meson.build b/src/test/modules/test_misc/meson.build
index 5d81f5b13be..ec41a7a373b 100644
--- a/src/test/modules/test_misc/meson.build
+++ b/src/test/modules/test_misc/meson.build
@@ -24,6 +24,7 @@ tests += {
't/013_temp_obj_multisession.pl',
't/014_log_statement_max_length.pl',
't/015_temp_schema_exit_deferrable.pl',
+ 't/016_copy_simd_eoc_marker.pl',
],
# The injection points are cluster-wide, so disable installcheck
'runningcheck': false,
diff --git a/src/test/modules/test_misc/t/016_copy_simd_eoc_marker.pl b/src/test/modules/test_misc/t/016_copy_simd_eoc_marker.pl
new file mode 100644
index 00000000000..31ce83e0b2c
--- /dev/null
+++ b/src/test/modules/test_misc/t/016_copy_simd_eoc_marker.pl
@@ -0,0 +1,90 @@
+
+# Copyright (c) 2021-2026, PostgreSQL Global Development Group
+
+# Regression test for a COPY FROM (FORMAT text) hang introduced by the SIMD
+# input scanner (commit e0a3a3fd536).
+#
+# CopyReadLineTextSIMDHelper() refills the input buffer once fewer than
+# sizeof(Vector8) bytes remain, before falling back to the scalar parser. If
+# those buffered bytes already hold a complete "\." end-of-copy marker and the
+# source is a still-open pipe with no further data ready, the refill blocks and
+# COPY fails to recognize the marker until the writer produces more data or
+# closes the pipe. The fix checks the short tail for a backslash and defers to
+# the scalar parser, which recognizes the marker without reading ahead.
+#
+# The writer program below emits exactly one full input buffer
+# (INPUT_BUF_SIZE = 65536 bytes): a long line, a complete "\." marker, and a
+# few trailing bytes so the marker lands in the short (< sizeof(Vector8)) tail.
+# It then keeps the pipe open with a slow dribble so a buggy refill blocks
+# (rather than seeing EOF) until statement_timeout would fire. With the fix,
+# COPY recognizes the marker immediately and the writer exits on SIGPIPE.
+
+use strict;
+use warnings FATAL => 'all';
+use File::Spec;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+if ($windows_os)
+{
+ plan skip_all =>
+ 'COPY FROM PROGRAM pipe-blocking scenario is unreliable on Windows';
+}
+
+# The writer script, run by COPY ... FROM PROGRAM. Single-quoted heredoc: the
+# contents are written to the file verbatim, so the "\." below reaches the
+# child perl as a literal backslash-dot end-of-copy marker.
+my $writer = <<'END_WRITER';
+$SIG{PIPE} = sub { exit 0 };
+my $data = ("a" x 65524) . "\n\\.\n" . ("x" x 8);
+die "payload is not 65536 bytes" unless length($data) == 65536;
+my $offset = 0;
+while ($offset < length($data))
+{
+ my $written = syswrite(STDOUT, $data, length($data) - $offset, $offset);
+ die "write failed: $!" unless defined $written;
+ $offset += $written;
+}
+while (1)
+{
+ select(undef, undef, undef, 0.05);
+ syswrite(STDOUT, "x");
+}
+END_WRITER
+
+# Absolute path: COPY ... FROM PROGRAM runs in the backend's data directory,
+# not the test's working directory.
+my $script = File::Spec->rel2abs(
+ "${PostgreSQL::Test::Utils::tmp_check}/copy_simd_eoc_writer.pl");
+append_to_file($script, $writer);
+
+my $node = PostgreSQL::Test::Cluster->new('primary');
+$node->init;
+$node->start;
+
+# statement_timeout bounds the failure: a buggy build blocks in the refill and
+# aborts here; a fixed build recognizes the marker and returns immediately,
+# independent of the timeout.
+my $sql = qq{
+SET statement_timeout = '30s';
+CREATE TEMP TABLE copy_simd_eoc (a text);
+COPY copy_simd_eoc FROM PROGRAM '$^X "$script"';
+SELECT count(*) || '|' || min(length(a)) FROM copy_simd_eoc;
+};
+
+my ($stdout, $stderr) = ('', '');
+my $ret = $node->psql(
+ 'postgres', $sql,
+ stdout => \$stdout,
+ stderr => \$stderr,
+ timeout => $PostgreSQL::Test::Utils::timeout_default);
+
+is($ret, 0,
+ 'COPY FROM pipe with a buffered end-of-copy marker does not block');
+is($stdout, '1|65524', 'exactly one row of the expected length was copied');
+diag("psql stderr: $stderr") if $ret != 0;
+
+$node->stop;
+
+done_testing();
--
2.50.1 (Apple Git-155)
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 22:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-10 11:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-21 20:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Neil Conway <neil.conway@gmail.com>
2026-01-31 16:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 13:51 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 18:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 21:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-06 22:19 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 22:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-11 13:27 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-11 22:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-13 11:45 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-13 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 13:38 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-18 21:26 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 21:54 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 04:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-19 09:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 22:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-20 00:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-20 09:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-20 18:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-23 09:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 04:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-24 13:57 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 17:48 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-25 14:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-02 19:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-04 15:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-09 18:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-10 12:35 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-10 17:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 11:36 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 18:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 18:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 19:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 19:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 20:42 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-12 10:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-12 17:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-13 11:57 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-13 13:34 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-13 14:05 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-13 15:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Greg Burd <greg@burd.me>
2026-03-13 16:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-13 17:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Greg Burd <greg@burd.me>
2026-03-13 17:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-08-31 14:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-08-31 17:48 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-09-01 04:41 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Greg Burd <greg@burd.me>
2026-09-01 08:17 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-09-01 19:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-09-07 16:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-09-08 10:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-09-08 15:08 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Greg Burd <greg@burd.me>
2026-09-08 15:36 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-09-08 15:58 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Greg Burd <greg@burd.me>
@ 2026-09-08 20:58 ` Nathan Bossart <nathandbossart@gmail.com>
2026-09-09 10:12 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
0 siblings, 1 reply; 178+ messages in thread
From: Nathan Bossart @ 2026-09-08 20:58 UTC (permalink / raw)
To: Greg Burd <greg@burd.me>; +Cc: Nazir Bilal Yavuz <byavuz81@gmail.com>; KAZAR Ayoub <ma_kazar@esi.dz>; Neil Conway <neil.conway@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
On Tue, Sep 08, 2026 at 11:58:42AM -0400, Greg Burd wrote:
> The test writes one full input buffer (INPUT_BUF_SIZE, 65536 bytes) down a
> COPY FROM PROGRAM pipe: a 65524-byte line, then a complete \. marker, then
> 8 trailing bytes. That leaves 11 bytes unexamined when the SIMD path wants
> to refill, which is under sizeof(Vector8) and already holds the whole
> marker. The writer then dribbles a byte every 50ms and keeps the pipe open,
> so a speculative load blocks rather than seeing EOF. statement_timeout
> bounds the failure, so a regression fails the test instead of hanging the
> suite. Skipped on Windows, where the blocking pipe and SIGPIPE behavior
> isn't reliable.
I'm not sure this particular corner case deserves its own TAP test. IMHO
that is disproportionately expensive. It really only serves to prevent us
from removing the backslash scan from the SIMD path, which I don't think
we're likely to do. Furthermore, I'm not aware of any existing examples
where a test writes and then invokes its own Perl script, and I can't get
too excited about being the first to commit something like that. So, I'd
prefer to leave this test out for now. If the new test fit cleanly into an
existing suite or was less magical, I might be more eager to include it,
but it's probably still unlikely to be worth the energy and maintenance
risk.
--
nathan
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 22:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-10 11:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-21 20:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Neil Conway <neil.conway@gmail.com>
2026-01-31 16:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 13:51 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 18:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 21:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-06 22:19 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 22:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-11 13:27 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-11 22:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-13 11:45 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-13 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 13:38 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-18 21:26 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 21:54 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 04:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-19 09:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 22:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-20 00:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-20 09:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-20 18:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-23 09:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 04:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-24 13:57 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 17:48 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-25 14:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-02 19:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-04 15:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-09 18:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-10 12:35 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-10 17:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 11:36 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 18:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 18:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 19:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 19:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 20:42 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-12 10:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-12 17:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-13 11:57 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-13 13:34 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-13 14:05 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-13 15:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Greg Burd <greg@burd.me>
2026-03-13 16:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-13 17:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Greg Burd <greg@burd.me>
2026-03-13 17:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-08-31 14:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-08-31 17:48 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-09-01 04:41 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Greg Burd <greg@burd.me>
2026-09-01 08:17 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-09-01 19:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-09-07 16:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-09-08 10:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-09-08 15:08 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Greg Burd <greg@burd.me>
2026-09-08 15:36 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-09-08 15:58 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Greg Burd <greg@burd.me>
2026-09-08 20:58 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
@ 2026-09-09 10:12 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-09-09 11:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Greg Burd <greg@burd.me>
0 siblings, 1 reply; 178+ messages in thread
From: Nazir Bilal Yavuz @ 2026-09-09 10:12 UTC (permalink / raw)
To: Nathan Bossart <nathandbossart@gmail.com>; +Cc: Greg Burd <greg@burd.me>; KAZAR Ayoub <ma_kazar@esi.dz>; Neil Conway <neil.conway@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
Hi,
On Tue, 8 Sept 2026 at 23:58, Nathan Bossart <nathandbossart@gmail.com> wrote:
>
> On Tue, Sep 08, 2026 at 11:58:42AM -0400, Greg Burd wrote:
> > The test writes one full input buffer (INPUT_BUF_SIZE, 65536 bytes) down a
> > COPY FROM PROGRAM pipe: a 65524-byte line, then a complete \. marker, then
> > 8 trailing bytes. That leaves 11 bytes unexamined when the SIMD path wants
> > to refill, which is under sizeof(Vector8) and already holds the whole
> > marker. The writer then dribbles a byte every 50ms and keeps the pipe open,
> > so a speculative load blocks rather than seeing EOF. statement_timeout
> > bounds the failure, so a regression fails the test instead of hanging the
> > suite. Skipped on Windows, where the blocking pipe and SIGPIPE behavior
> > isn't reliable.
>
> I'm not sure this particular corner case deserves its own TAP test.
I have similar feelings.
> IMHO
> that is disproportionately expensive. It really only serves to prevent us
> from removing the backslash scan from the SIMD path, which I don't think
> we're likely to do.
I too thought it would be expensive but it wasn't on my machine; it
took 0.5s, which is lower than I expected.
> Furthermore, I'm not aware of any existing examples
> where a test writes and then invokes its own Perl script, and I can't get
> too excited about being the first to commit something like that. So, I'd
> prefer to leave this test out for now. If the new test fit cleanly into an
> existing suite or was less magical, I might be more eager to include it,
> but it's probably still unlikely to be worth the energy and maintenance
> risk.
Perhaps we can save this Perl script as another file and invoke that
file. This seems better to me. However, I agree that adding this test
might be overkill given the current circumstances.
--
Regards,
Nazir Bilal Yavuz
Microsoft
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 22:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-10 11:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-21 20:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Neil Conway <neil.conway@gmail.com>
2026-01-31 16:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 13:51 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 18:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 21:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-06 22:19 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 22:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-11 13:27 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-11 22:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-13 11:45 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-13 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 13:38 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-18 21:26 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 21:54 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 04:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-19 09:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 22:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-20 00:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-20 09:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-20 18:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-23 09:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 04:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-24 13:57 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 17:48 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-25 14:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-02 19:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-04 15:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-09 18:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-10 12:35 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-10 17:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 11:36 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 18:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 18:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 19:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 19:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 20:42 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-12 10:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-12 17:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-13 11:57 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-13 13:34 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-13 14:05 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-13 15:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Greg Burd <greg@burd.me>
2026-03-13 16:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-13 17:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Greg Burd <greg@burd.me>
2026-03-13 17:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-08-31 14:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-08-31 17:48 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-09-01 04:41 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Greg Burd <greg@burd.me>
2026-09-01 08:17 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-09-01 19:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-09-07 16:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-09-08 10:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-09-08 15:08 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Greg Burd <greg@burd.me>
2026-09-08 15:36 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-09-08 15:58 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Greg Burd <greg@burd.me>
2026-09-08 20:58 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-09-09 10:12 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
@ 2026-09-09 11:14 ` Greg Burd <greg@burd.me>
2026-09-09 21:07 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
0 siblings, 1 reply; 178+ messages in thread
From: Greg Burd @ 2026-09-09 11:14 UTC (permalink / raw)
To: Nazir Bilal Yavuz <byavuz81@gmail.com>; Nathan Bossart <nathandbossart@gmail.com>; +Cc: KAZAR Ayoub <ma_kazar@esi.dz>; Neil Conway <neil.conway@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
On Wed, Sep 9, 2026, at 6:12 AM, Nazir Bilal Yavuz wrote:
> Hi,
>
> On Tue, 8 Sept 2026 at 23:58, Nathan Bossart <nathandbossart@gmail.com> wrote:
>>
>> On Tue, Sep 08, 2026 at 11:58:42AM -0400, Greg Burd wrote:
>> > The test writes one full input buffer (INPUT_BUF_SIZE, 65536 bytes) down a
>> > COPY FROM PROGRAM pipe: a 65524-byte line, then a complete \. marker, then
>> > 8 trailing bytes. That leaves 11 bytes unexamined when the SIMD path wants
>> > to refill, which is under sizeof(Vector8) and already holds the whole
>> > marker. The writer then dribbles a byte every 50ms and keeps the pipe open,
>> > so a speculative load blocks rather than seeing EOF. statement_timeout
>> > bounds the failure, so a regression fails the test instead of hanging the
>> > suite. Skipped on Windows, where the blocking pipe and SIGPIPE behavior
>> > isn't reliable.
>>
>> I'm not sure this particular corner case deserves its own TAP test.
>
> I have similar feelings.
It was just a first swing at a test, and I agree with both of you it felt
heavyweight for the issue at hand. But it does capture the issue.
>> IMHO
>> that is disproportionately expensive. It really only serves to prevent us
>> from removing the backslash scan from the SIMD path, which I don't think
>> we're likely to do.
>
> I too thought it would be expensive but it wasn't on my machine; it
> took 0.5s, which is lower than I expected.
>
>
>> Furthermore, I'm not aware of any existing examples
>> where a test writes and then invokes its own Perl script, and I can't get
>> too excited about being the first to commit something like that. So, I'd
>> prefer to leave this test out for now. If the new test fit cleanly into an
>> existing suite or was less magical, I might be more eager to include it,
>> but it's probably still unlikely to be worth the energy and maintenance
>> risk.
Yeah, that's a solid judgement call. One I should have made myself.
> Perhaps we can save this Perl script as another file and invoke that
> file. This seems better to me. However, I agree that adding this test
> might be overkill given the current circumstances.
It's overkill, let's focus and get the fix in.
> --
> Regards,
> Nazir Bilal Yavuz
> Microsoft
best.
-greg
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 22:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-10 11:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-21 20:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Neil Conway <neil.conway@gmail.com>
2026-01-31 16:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 13:51 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 18:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 21:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-06 22:19 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 22:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-11 13:27 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-11 22:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-13 11:45 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-13 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 13:38 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-18 21:26 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 21:54 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 04:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-19 09:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 22:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-20 00:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-20 09:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-20 18:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-23 09:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 04:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-24 13:57 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 17:48 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-25 14:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-02 19:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-04 15:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-09 18:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-10 12:35 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-10 17:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 11:36 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 18:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 18:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 19:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 19:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 20:42 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-12 10:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-12 17:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-13 11:57 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-13 13:34 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-13 14:05 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-13 15:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Greg Burd <greg@burd.me>
2026-03-13 16:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-13 17:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Greg Burd <greg@burd.me>
2026-03-13 17:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-08-31 14:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-08-31 17:48 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-09-01 04:41 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Greg Burd <greg@burd.me>
2026-09-01 08:17 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-09-01 19:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-09-07 16:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-09-08 10:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-09-08 15:08 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Greg Burd <greg@burd.me>
2026-09-08 15:36 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-09-08 15:58 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Greg Burd <greg@burd.me>
2026-09-08 20:58 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-09-09 10:12 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-09-09 11:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Greg Burd <greg@burd.me>
@ 2026-09-09 21:07 ` Nathan Bossart <nathandbossart@gmail.com>
0 siblings, 0 replies; 178+ messages in thread
From: Nathan Bossart @ 2026-09-09 21:07 UTC (permalink / raw)
To: Greg Burd <greg@burd.me>; +Cc: Nazir Bilal Yavuz <byavuz81@gmail.com>; KAZAR Ayoub <ma_kazar@esi.dz>; Neil Conway <neil.conway@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
> It's overkill, let's focus and get the fix in.
Committed.
--
nathan
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 22:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-10 11:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-21 20:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Neil Conway <neil.conway@gmail.com>
2026-01-31 16:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 13:51 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 18:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 21:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-06 22:19 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 22:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-11 13:27 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-11 22:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-13 11:45 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-13 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 13:38 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-18 21:26 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 21:54 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 04:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-19 09:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 22:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-20 00:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-20 09:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-20 18:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-23 09:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 04:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-24 13:57 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 17:48 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-25 14:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-02 19:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-04 15:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-09 18:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-10 12:35 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-10 17:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 11:36 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 18:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 18:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 19:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 19:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 20:42 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-12 10:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-12 17:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-13 11:57 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-13 13:34 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-13 14:05 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-13 15:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Greg Burd <greg@burd.me>
2026-03-13 16:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-13 17:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Greg Burd <greg@burd.me>
2026-03-13 17:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-08-31 14:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-08-31 17:48 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-09-01 04:41 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Greg Burd <greg@burd.me>
2026-09-01 08:17 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-09-01 19:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-09-07 16:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-09-08 10:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
@ 2026-09-08 15:24 ` Nathan Bossart <nathandbossart@gmail.com>
2026-09-09 10:00 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
1 sibling, 1 reply; 178+ messages in thread
From: Nathan Bossart @ 2026-09-08 15:24 UTC (permalink / raw)
To: Nazir Bilal Yavuz <byavuz81@gmail.com>; +Cc: Greg Burd <greg@burd.me>; KAZAR Ayoub <ma_kazar@esi.dz>; Neil Conway <neil.conway@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
On Tue, Sep 08, 2026 at 01:11:00PM +0300, Nazir Bilal Yavuz wrote:
> Here is an attempt to solve this problem. I ran Manni's script and saw
> a 1-2% slowdown on TEXT-wide inputs. It is still faster compared to
> the version without the SIMD patch; the slowdown is relative to the
> current master branch (which includes the SIMD patch).
Thanks for reporting. Fable 5.1 suggests moving the check to
CopyLoadInputBuf() to avoid this small regression, which makes sense to me.
I've attached a new patch that does that. This is definitely a bit of
hack, and I'm assuming we'll need to revisit this if we want to re-attempt
more sophisticated heuristics for when to use the SIMD path. But IMHO this
is self-contained enough to be reasonable for v19.
--
nathan
From 2323f8bbc97467238859ece05f3b6da5524c7593 Mon Sep 17 00:00:00 2001
From: Nathan Bossart <nathan@postgresql.org>
Date: Tue, 8 Sep 2026 10:24:31 -0500
Subject: [PATCH v2 1/1] Fix hangs in COPY FROM (FORMAT text).
The SIMD path for this command reads ahead via CopyLoadInputBuf()
whenever fewer than sizeof(Vector8) bytes remain in the input
buffer, even if those bytes hold a complete end-of-copy marker.
If the input is a pipe whose writer has sent the marker but not
closed the pipe, that read blocks, and COPY waits for data it will
never use. The scalar loop asks only for the bytes it needs, so
it stops at the marker without reading any further.
To fix, teach CopyLoadInputBuf() to decline a speculative load
when the caller's remaining bytes contain a backslash, which in
text mode might begin such a marker. (CSV mode doesn't treat
\. as special, so it is unaffected.) The SIMD path then hands
those bytes to the scalar loop, as it already does when a load
comes up short. Checking for the marker in the SIMD helper itself
would be more direct, but a call there costs the compiler
registers on every line, which measurably slowed COPY of short
lines in my testing.
Oversight in commit e0a3a3fd53.
Author: Nazir Bilal Yavuz <byavuz81@gmail.com>
Discussion: https://postgr.es/m/CAN55FZ1qFb4Yo3-MWeQfGQnu1_Ksx4xcFC2m3hyw8a--%2BeHQYQ%40mail.gmail.com
Backpatch-through: 19
---
src/backend/commands/copyfromparse.c | 22 ++++++++++++++++++----
1 file changed, 18 insertions(+), 4 deletions(-)
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index 98bf30ef2e7..68088108c7c 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -652,16 +652,30 @@ CopyLoadRawBuf(CopyFromState cstate)
* If INPUT_BUF_BYTES(cstate) > 0, the unprocessed bytes are moved to the start
* of the buffer and then we load more data after that.
*
- * If "speculative" is true, this function skips reporting any encoding or
- * conversion errors, provided there are still data for the caller to process.
- * Such callers must be prepared for this function to return without loading
- * anything.
+ * If "speculative" is true, the caller has not yet examined the data left in
+ * input_buf and needn't load more to make progress. In that case, this
+ * function skips reporting any encoding or conversion errors, and in text
+ * mode it declines to read past a backslash, as that might begin an
+ * end-of-copy marker. Such callers must be prepared for this function to
+ * return without loading anything.
*/
static void
CopyLoadInputBuf(CopyFromState cstate, bool speculative)
{
int nbytes = INPUT_BUF_BYTES(cstate);
+ /*
+ * In text mode, a backslash among the bytes a speculative caller has yet
+ * to examine might begin an end-of-copy marker. The caller must find
+ * that on its own, without waiting on input that may never arrive, as
+ * from a pipe whose writer has sent the marker but not closed the pipe.
+ * So in that case, just return without loading anything.
+ */
+ if (speculative && cstate->opts.format == COPY_FORMAT_TEXT &&
+ memchr(cstate->input_buf + cstate->input_buf_index, '\\',
+ nbytes) != NULL)
+ return;
+
/*
* The caller has updated input_buf_index to indicate how much of the
* input has been consumed and isn't needed anymore. If input_buf is the
--
2.55.0
Attachments:
[text/plain] v2-0001-Fix-hangs-in-COPY-FROM-FORMAT-text.patch (3.3K, ../../aqAoyyC1Qlx62KQ0@nathan/2-v2-0001-Fix-hangs-in-COPY-FROM-FORMAT-text.patch)
download | inline diff:
From 2323f8bbc97467238859ece05f3b6da5524c7593 Mon Sep 17 00:00:00 2001
From: Nathan Bossart <nathan@postgresql.org>
Date: Tue, 8 Sep 2026 10:24:31 -0500
Subject: [PATCH v2 1/1] Fix hangs in COPY FROM (FORMAT text).
The SIMD path for this command reads ahead via CopyLoadInputBuf()
whenever fewer than sizeof(Vector8) bytes remain in the input
buffer, even if those bytes hold a complete end-of-copy marker.
If the input is a pipe whose writer has sent the marker but not
closed the pipe, that read blocks, and COPY waits for data it will
never use. The scalar loop asks only for the bytes it needs, so
it stops at the marker without reading any further.
To fix, teach CopyLoadInputBuf() to decline a speculative load
when the caller's remaining bytes contain a backslash, which in
text mode might begin such a marker. (CSV mode doesn't treat
\. as special, so it is unaffected.) The SIMD path then hands
those bytes to the scalar loop, as it already does when a load
comes up short. Checking for the marker in the SIMD helper itself
would be more direct, but a call there costs the compiler
registers on every line, which measurably slowed COPY of short
lines in my testing.
Oversight in commit e0a3a3fd53.
Author: Nazir Bilal Yavuz <byavuz81@gmail.com>
Discussion: https://postgr.es/m/CAN55FZ1qFb4Yo3-MWeQfGQnu1_Ksx4xcFC2m3hyw8a--%2BeHQYQ%40mail.gmail.com
Backpatch-through: 19
---
src/backend/commands/copyfromparse.c | 22 ++++++++++++++++++----
1 file changed, 18 insertions(+), 4 deletions(-)
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index 98bf30ef2e7..68088108c7c 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -652,16 +652,30 @@ CopyLoadRawBuf(CopyFromState cstate)
* If INPUT_BUF_BYTES(cstate) > 0, the unprocessed bytes are moved to the start
* of the buffer and then we load more data after that.
*
- * If "speculative" is true, this function skips reporting any encoding or
- * conversion errors, provided there are still data for the caller to process.
- * Such callers must be prepared for this function to return without loading
- * anything.
+ * If "speculative" is true, the caller has not yet examined the data left in
+ * input_buf and needn't load more to make progress. In that case, this
+ * function skips reporting any encoding or conversion errors, and in text
+ * mode it declines to read past a backslash, as that might begin an
+ * end-of-copy marker. Such callers must be prepared for this function to
+ * return without loading anything.
*/
static void
CopyLoadInputBuf(CopyFromState cstate, bool speculative)
{
int nbytes = INPUT_BUF_BYTES(cstate);
+ /*
+ * In text mode, a backslash among the bytes a speculative caller has yet
+ * to examine might begin an end-of-copy marker. The caller must find
+ * that on its own, without waiting on input that may never arrive, as
+ * from a pipe whose writer has sent the marker but not closed the pipe.
+ * So in that case, just return without loading anything.
+ */
+ if (speculative && cstate->opts.format == COPY_FORMAT_TEXT &&
+ memchr(cstate->input_buf + cstate->input_buf_index, '\\',
+ nbytes) != NULL)
+ return;
+
/*
* The caller has updated input_buf_index to indicate how much of the
* input has been consumed and isn't needed anymore. If input_buf is the
--
2.55.0
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 22:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-10 11:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-21 20:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Neil Conway <neil.conway@gmail.com>
2026-01-31 16:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 13:51 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 18:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 21:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-06 22:19 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 22:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-11 13:27 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-11 22:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-13 11:45 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-13 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 13:38 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-18 21:26 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 21:54 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 04:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-19 09:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 22:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-20 00:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-20 09:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-20 18:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-23 09:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 04:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-24 13:57 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 17:48 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-25 14:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-02 19:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-04 15:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-09 18:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-10 12:35 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-10 17:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 11:36 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 18:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 18:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 19:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 19:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 20:42 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-12 10:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-12 17:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-13 11:57 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-13 13:34 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-13 14:05 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-13 15:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Greg Burd <greg@burd.me>
2026-03-13 16:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-13 17:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Greg Burd <greg@burd.me>
2026-03-13 17:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-08-31 14:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-08-31 17:48 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-09-01 04:41 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Greg Burd <greg@burd.me>
2026-09-01 08:17 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-09-01 19:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-09-07 16:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-09-08 10:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-09-08 15:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
@ 2026-09-09 10:00 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
0 siblings, 0 replies; 178+ messages in thread
From: Nazir Bilal Yavuz @ 2026-09-09 10:00 UTC (permalink / raw)
To: Nathan Bossart <nathandbossart@gmail.com>; +Cc: Greg Burd <greg@burd.me>; KAZAR Ayoub <ma_kazar@esi.dz>; Neil Conway <neil.conway@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
Hi,
On Tue, 8 Sept 2026 at 18:25, Nathan Bossart <nathandbossart@gmail.com> wrote:
>
> On Tue, Sep 08, 2026 at 01:11:00PM +0300, Nazir Bilal Yavuz wrote:
> > Here is an attempt to solve this problem. I ran Manni's script and saw
> > a 1-2% slowdown on TEXT-wide inputs. It is still faster compared to
> > the version without the SIMD patch; the slowdown is relative to the
> > current master branch (which includes the SIMD patch).
>
> Thanks for reporting. Fable 5.1 suggests moving the check to
> CopyLoadInputBuf() to avoid this small regression, which makes sense to me.
> I've attached a new patch that does that. This is definitely a bit of
> hack, and I'm assuming we'll need to revisit this if we want to re-attempt
> more sophisticated heuristics for when to use the SIMD path. But IMHO this
> is self-contained enough to be reasonable for v19.
Thanks for v2; I agree that it is better. I confirm that it solves the
problem and doesn't cause the 1-2% regression I mentioned.
--
Regards,
Nazir Bilal Yavuz
Microsoft
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 22:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-10 11:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-21 20:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Neil Conway <neil.conway@gmail.com>
2026-01-31 16:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 13:51 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 18:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 21:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-06 22:19 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 22:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-11 13:27 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-11 22:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-13 11:45 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-13 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 13:38 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-18 21:26 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 21:54 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 04:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-19 09:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 22:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-20 00:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-20 09:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-20 18:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-23 09:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 04:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-24 13:57 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 17:48 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-25 14:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-02 19:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-04 15:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-09 18:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-10 12:35 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-10 17:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 11:36 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 18:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 18:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 19:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 19:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 20:42 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-12 10:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-12 17:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-13 11:57 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-13 13:34 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-13 14:05 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
@ 2026-03-13 15:58 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-13 16:08 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
1 sibling, 1 reply; 178+ messages in thread
From: Nazir Bilal Yavuz @ 2026-03-13 15:58 UTC (permalink / raw)
To: Nathan Bossart <nathandbossart@gmail.com>; +Cc: Manni Wood <manni.wood@enterprisedb.com>; KAZAR Ayoub <ma_kazar@esi.dz>; Neil Conway <neil.conway@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
Hi,
On Fri, 13 Mar 2026 at 17:05, Nathan Bossart <nathandbossart@gmail.com>
wrote:
>
> On Fri, Mar 13, 2026 at 04:34:49PM +0300, Nazir Bilal Yavuz wrote:
> > On Fri, 13 Mar 2026 at 14:57, Nazir Bilal Yavuz <byavuz81@gmail.com>
wrote:
> >> Unfortunately, v15 causes a regression for a 'csv & wide & 1/3' case
> >> on my end. v14 was taking 8000ms but v15 took ~9100ms. If we add the
> >> tmp_hit_eof variable then the regression disappears. Also, if I use a
> >> struct like below, regression disappears again.
> >
> >> When I removed the tmp_hit_eof variable on v14, I didn't encounter any
> >> regression. I really don't understand why this is happening on my end.
> >> Manni didn't encounter any regression on the benchmark [1].
> >
> > Problem might be related to gcc. I am using Debian Trixie and my
> > current gcc version is 'gcc version 14.2.0 (Debian 14.2.0-19)'. If I
> > compile Postgres with 'Debian clang version 19.1.7 (3+b1)', then there
> > is no regression, which makes more sense IMO.
>
> Let's just re-add the temporary variable for hit_eof. The struct idea is
> clever, but it's just a little more complicated than I think is necessary
> here.
>
> I've also removed the goto in favor of just duplicating the "out" code,
> like you had before. I'd like to avoid sporadic #ifndef USE_NO_SIMD uses,
> and goto is out of fashion, anyway.
Thanks! v17 LGTM. I didn't encounter any regressions.
--
Regards,
Nazir Bilal Yavuz
Microsoft
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 22:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-10 11:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-21 20:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Neil Conway <neil.conway@gmail.com>
2026-01-31 16:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 13:51 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 18:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 21:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-06 22:19 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 22:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-11 13:27 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-11 22:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-13 11:45 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-13 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 13:38 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-18 21:26 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 21:54 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 04:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-19 09:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 22:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-20 00:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-20 09:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-20 18:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-23 09:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 04:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-24 13:57 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 17:48 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-25 14:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-02 19:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-04 15:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-09 18:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-10 12:35 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-10 17:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 11:36 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 18:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 18:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 19:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 19:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 20:42 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-12 10:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-12 17:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-13 11:57 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-13 13:34 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-13 14:05 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-13 15:58 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
@ 2026-03-13 16:08 ` Nathan Bossart <nathandbossart@gmail.com>
2026-03-13 16:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
0 siblings, 1 reply; 178+ messages in thread
From: Nathan Bossart @ 2026-03-13 16:08 UTC (permalink / raw)
To: Nazir Bilal Yavuz <byavuz81@gmail.com>; +Cc: Manni Wood <manni.wood@enterprisedb.com>; KAZAR Ayoub <ma_kazar@esi.dz>; Neil Conway <neil.conway@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
On Fri, Mar 13, 2026 at 06:58:49PM +0300, Nazir Bilal Yavuz wrote:
> Thanks! v17 LGTM. I didn't encounter any regressions.
Committed.
--
nathan
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 22:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-10 11:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-21 20:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Neil Conway <neil.conway@gmail.com>
2026-01-31 16:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 13:51 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 18:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 21:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-06 22:19 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 22:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-11 13:27 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-11 22:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-13 11:45 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-13 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 13:38 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-18 21:26 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 21:54 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 04:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-19 09:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 22:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-20 00:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-20 09:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-20 18:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-23 09:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 04:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-24 13:57 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 17:48 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-25 14:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-02 19:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-04 15:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-09 18:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-10 12:35 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-10 17:10 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 11:36 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 18:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 18:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 19:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 19:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 20:42 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-12 10:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-12 17:37 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-13 11:57 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-13 13:34 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-13 14:05 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-03-13 15:58 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-13 16:08 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
@ 2026-03-13 16:13 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
0 siblings, 0 replies; 178+ messages in thread
From: Nazir Bilal Yavuz @ 2026-03-13 16:13 UTC (permalink / raw)
To: Nathan Bossart <nathandbossart@gmail.com>; +Cc: Manni Wood <manni.wood@enterprisedb.com>; KAZAR Ayoub <ma_kazar@esi.dz>; Neil Conway <neil.conway@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
Hi,
On Fri, 13 Mar 2026 at 19:08, Nathan Bossart <nathandbossart@gmail.com> wrote:
>
> On Fri, Mar 13, 2026 at 06:58:49PM +0300, Nazir Bilal Yavuz wrote:
> > Thanks! v17 LGTM. I didn't encounter any regressions.
>
> Committed.
Thank you for taking care of this!
--
Regards,
Nazir Bilal Yavuz
Microsoft
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 22:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-10 11:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-21 20:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Neil Conway <neil.conway@gmail.com>
2026-01-31 16:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 13:51 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 18:11 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 21:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-06 22:19 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 22:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-11 13:27 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-11 22:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2026-02-13 11:45 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-13 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
@ 2026-02-20 10:01 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2 siblings, 0 replies; 178+ messages in thread
From: Nazir Bilal Yavuz @ 2026-02-20 10:01 UTC (permalink / raw)
To: Nathan Bossart <nathandbossart@gmail.com>; +Cc: KAZAR Ayoub <ma_kazar@esi.dz>; Neil Conway <neil.conway@gmail.com>; Manni Wood <manni.wood@enterprisedb.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
Hi,
On Sat, 14 Feb 2026 at 02:09, Nathan Bossart <nathandbossart@gmail.com> wrote:
>
> Some other random thoughts:
>
> + match = vector8_or(vector8_eq(chunk, nl), vector8_eq(chunk, cr));
>
> + match = vector8_or(vector8_eq(chunk, nl), vector8_eq(chunk, cr));
>
> Since \n and \r are well below "normal" ASCII values, I wonder if we could
> simplify these to something like
>
> match = vector8_gt(... vector with all lanes set to \r + 1 ..., chunk);
This didn't work because we have horizontal tab characters whose value
(9) is lower than '\r' (13).
> + /* Check if we found any special characters */
> + mask = vector8_highbit_mask(match);
> + if (mask != 0)
>
> vector8_highbit_mask() is somewhat expensive on AArch64, so I wonder if
> waiting until we enter the "if" block to calculate it has any benefit.
I think this makes sense, done.
> + simd_hit_eol = (c1 == '\r' || c1 == '\n') && (!is_csv || !in_quote);
>
> If (is_csv && in_quote), we shouldn't have picked up \r or \n in the first
> place, right?
You are right, I put an assertion for this.
> + simd_hit_eof = c1 == '\\' && c2 == '.' && !is_csv;
> +
> + /*
> + * Do not disable SIMD when we hit EOL or EOF characters. In
> + * practice, it does not matter for EOF because parsing ends
> + * there, but we keep the behavior consistent.
> + */
> + if (!(simd_hit_eof || simd_hit_eol))
>
> I'd think that doing less unnecessary work would outweigh the benefits of
> consistency for the EOF case.
This will work once for the data since SIMD will be disabled
afterwards. So, I think this shouldn't affect the performance but I am
okay to change if you prefer.
I have bencharmed v9 and didn't see any regression.
--
Regards,
Nazir Bilal Yavuz
Microsoft
Attachments:
[text/x-patch] v9-0001-Speedup-COPY-FROM-with-additional-function-inlini.patch (2.6K, ../../CAN55FZ0MiFCgK26gRgE05a=_ggenkxDM8H=A2uTHpywczqt=-Q@mail.gmail.com/2-v9-0001-Speedup-COPY-FROM-with-additional-function-inlini.patch)
download | inline diff:
From 05da2c50de603bd4c2103e8a0c8a37af0599c1f9 Mon Sep 17 00:00:00 2001
From: Nathan Bossart <nathan@postgresql.org>
Date: Wed, 18 Feb 2026 14:55:39 -0600
Subject: [PATCH v9 1/2] Speedup COPY FROM with additional function inlining.
Following the example set by commit 58a359e585, we can squeeze out
a little more performance from COPY FROM (FORMAT {text,csv}) by
forcing CopyReadLineText() to be inlined and by passing the is_csv
parameter as a constant. This allows the compiler to emit
specialized code with the known-const false comparisons and
subsequent branches removed.
Author: Nazir Bilal Yavuz <byavuz81@gmail.com>
Reviewed-by: Ayoub Kazar <ma_kazar@esi.dz>
Discussion: https://postgr.es/m/CAOzEurSW8cNr6TPKsjrstnPfhf4QyQqB4tnPXGGe8N4e_v7Jig%40mail.gmail.com
---
src/backend/commands/copyfromparse.c | 19 +++++++++++++++----
1 file changed, 15 insertions(+), 4 deletions(-)
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index 94d6f415a06..0aa549837b5 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -141,7 +141,8 @@ static const char BinarySignature[11] = "PGCOPY\n\377\r\n\0";
/* non-export function prototypes */
static bool CopyReadLine(CopyFromState cstate, bool is_csv);
-static bool CopyReadLineText(CopyFromState cstate, bool is_csv);
+static pg_attribute_always_inline bool CopyReadLineText(CopyFromState cstate,
+ bool is_csv);
static int CopyReadAttributesText(CopyFromState cstate);
static int CopyReadAttributesCSV(CopyFromState cstate);
static Datum CopyReadBinaryAttribute(CopyFromState cstate, FmgrInfo *flinfo,
@@ -1173,8 +1174,18 @@ CopyReadLine(CopyFromState cstate, bool is_csv)
resetStringInfo(&cstate->line_buf);
cstate->line_buf_valid = false;
- /* Parse data and transfer into line_buf */
- result = CopyReadLineText(cstate, is_csv);
+ /*
+ * Parse data and transfer into line_buf.
+ *
+ * Because this is performance critical, we inline CopyReadLineText() and
+ * pass the boolean parameters as constants to allow the compiler to emit
+ * specialized code with the known-const false comparisons and subsequent
+ * branches removed.
+ */
+ if (is_csv)
+ result = CopyReadLineText(cstate, true);
+ else
+ result = CopyReadLineText(cstate, false);
if (result)
{
@@ -1241,7 +1252,7 @@ CopyReadLine(CopyFromState cstate, bool is_csv)
/*
* CopyReadLineText - inner loop of CopyReadLine for text mode
*/
-static bool
+static pg_attribute_always_inline bool
CopyReadLineText(CopyFromState cstate, bool is_csv)
{
char *copy_input_buf;
--
2.47.3
[text/x-patch] v9-0002-Speed-up-COPY-FROM-text-CSV-parsing-using-SIMD.patch (7.7K, ../../CAN55FZ0MiFCgK26gRgE05a=_ggenkxDM8H=A2uTHpywczqt=-Q@mail.gmail.com/3-v9-0002-Speed-up-COPY-FROM-text-CSV-parsing-using-SIMD.patch)
download | inline diff:
From 62d350341ab933d6c0bc3d6a72bc3d1aed36c205 Mon Sep 17 00:00:00 2001
From: Nazir Bilal Yavuz <byavuz81@gmail.com>
Date: Fri, 13 Feb 2026 13:28:55 +0300
Subject: [PATCH v9 2/2] Speed up COPY FROM text/CSV parsing using SIMD
This patch disables SIMD when SIMD encounters a special character which
is neither EOF nor EOL.
Author: Shinya Kato <shinya11.kato@gmail.com>
Author: Nazir Bilal Yavuz <byavuz81@gmail.com>
Reviewed-by: Kazar Ayoub <ma_kazar@esi.dz>
Reviewed-by: Nathan Bossart <nathandbossart@gmail.com>
Reviewed-by: Neil Conway <neil.conway@gmail.com>
Reviewed-by: Andrew Dunstan <andrew@dunslane.net>
Reviewed-by: Manni Wood <manni.wood@enterprisedb.com>
Reviewed-by: Mark Wong <markwkm@gmail.com>
Discussion: https://postgr.es/m/CAOzEurSW8cNr6TPKsjrstnPfhf4QyQqB4tnPXGGe8N4e_v7Jig%40mail.gmail.com
---
src/backend/commands/copyfrom.c | 3 +
src/backend/commands/copyfromparse.c | 135 ++++++++++++++++++++++-
src/include/commands/copyfrom_internal.h | 3 +
3 files changed, 137 insertions(+), 4 deletions(-)
diff --git a/src/backend/commands/copyfrom.c b/src/backend/commands/copyfrom.c
index 2b7556b287c..3dd159f15b2 100644
--- a/src/backend/commands/copyfrom.c
+++ b/src/backend/commands/copyfrom.c
@@ -1717,6 +1717,9 @@ BeginCopyFrom(ParseState *pstate,
cstate->cur_attval = NULL;
cstate->relname_only = false;
+ /* Initialize SIMD */
+ cstate->simd_enabled = true;
+
/*
* Allocate buffers for the input pipeline.
*
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index 0aa549837b5..7ca261a6def 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -72,6 +72,7 @@
#include "miscadmin.h"
#include "pgstat.h"
#include "port/pg_bswap.h"
+#include "port/simd.h"
#include "utils/builtins.h"
#include "utils/rel.h"
@@ -142,7 +143,8 @@ static const char BinarySignature[11] = "PGCOPY\n\377\r\n\0";
/* non-export function prototypes */
static bool CopyReadLine(CopyFromState cstate, bool is_csv);
static pg_attribute_always_inline bool CopyReadLineText(CopyFromState cstate,
- bool is_csv);
+ bool is_csv,
+ bool simd_enabled);
static int CopyReadAttributesText(CopyFromState cstate);
static int CopyReadAttributesCSV(CopyFromState cstate);
static Datum CopyReadBinaryAttribute(CopyFromState cstate, FmgrInfo *flinfo,
@@ -1183,9 +1185,19 @@ CopyReadLine(CopyFromState cstate, bool is_csv)
* branches removed.
*/
if (is_csv)
- result = CopyReadLineText(cstate, true);
+ {
+ if (cstate->simd_enabled)
+ result = CopyReadLineText(cstate, true, true);
+ else
+ result = CopyReadLineText(cstate, true, false);
+ }
else
- result = CopyReadLineText(cstate, false);
+ {
+ if (cstate->simd_enabled)
+ result = CopyReadLineText(cstate, false, true);
+ else
+ result = CopyReadLineText(cstate, false, false);
+ }
if (result)
{
@@ -1253,7 +1265,7 @@ CopyReadLine(CopyFromState cstate, bool is_csv)
* CopyReadLineText - inner loop of CopyReadLine for text mode
*/
static pg_attribute_always_inline bool
-CopyReadLineText(CopyFromState cstate, bool is_csv)
+CopyReadLineText(CopyFromState cstate, bool is_csv, bool simd_enabled)
{
char *copy_input_buf;
int input_buf_ptr;
@@ -1268,6 +1280,14 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
char quotec = '\0';
char escapec = '\0';
+#ifndef USE_NO_SIMD
+ Vector8 nl = vector8_broadcast('\n');
+ Vector8 cr = vector8_broadcast('\r');
+ Vector8 bs = vector8_broadcast('\\');
+ Vector8 quote = vector8_broadcast(0);
+ Vector8 escape = vector8_broadcast(0);
+#endif
+
if (is_csv)
{
quotec = cstate->opts.quote[0];
@@ -1275,6 +1295,12 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
/* ignore special escape processing if it's the same as quotec */
if (quotec == escapec)
escapec = '\0';
+
+#ifndef USE_NO_SIMD
+ quote = vector8_broadcast(quotec);
+ if (quotec != escapec)
+ escape = vector8_broadcast(escapec);
+#endif
}
/*
@@ -1341,6 +1367,107 @@ CopyReadLineText(CopyFromState cstate, bool is_csv)
need_data = false;
}
+#ifndef USE_NO_SIMD
+
+ /*
+ * Use SIMD instructions to efficiently scan the input buffer for
+ * special characters (e.g., newline, carriage return, quote, and
+ * escape). This is faster than byte-by-byte iteration, especially on
+ * large buffers.
+ *
+ * We do not apply the SIMD fast path in either of the following
+ * cases: - When the previously processed character was an escape
+ * character (last_was_esc), since the next byte must be examined
+ * sequentially. - When the remaining buffer is smaller than one
+ * vector width (sizeof(Vector8)), since SIMD operates on fixed-size
+ * chunks.
+ *
+ * Note that, SIMD may become slower when the input contains many
+ * special characters. To avoid this regression, we disable SIMD for
+ * the rest of the input once we encounter a special character which
+ * is neither EOF nor EOL.
+ */
+ if (simd_enabled && !last_was_esc && copy_buf_len - input_buf_ptr > sizeof(Vector8))
+ {
+ Vector8 chunk;
+ Vector8 match = vector8_broadcast(0);
+
+ /* Load a chunk of data into a vector register */
+ vector8_load(&chunk, (const uint8 *) ©_input_buf[input_buf_ptr]);
+
+ if (is_csv)
+ {
+ /* \n and \r are not special inside quotes */
+ if (!in_quote)
+ match = vector8_or(vector8_eq(chunk, nl), vector8_eq(chunk, cr));
+
+ match = vector8_or(match, vector8_eq(chunk, quote));
+ if (escapec != '\0')
+ match = vector8_or(match, vector8_eq(chunk, escape));
+ }
+ else
+ {
+ match = vector8_or(vector8_eq(chunk, nl), vector8_eq(chunk, cr));
+ match = vector8_or(match, vector8_eq(chunk, bs));
+ }
+
+ /* Check if we found any special characters */
+ if (vector8_is_highbit_set(match))
+ {
+ /*
+ * Found a special character. Advance up to that point and let
+ * the scalar code handle it.
+ */
+ uint32 mask;
+ int advance;
+ char c1,
+ c2;
+ bool simd_hit_eol,
+ simd_hit_eof;
+
+ mask = vector8_highbit_mask(match);
+ advance = pg_rightmost_one_pos32(mask);
+
+ input_buf_ptr += advance;
+ c1 = copy_input_buf[input_buf_ptr];
+
+ /*
+ * Since we stopped within the chunk and ((copy_buf_len -
+ * input_buf_ptr) > sizeof(Vector8)) is true,
+ * copy_input_buf[input_buf_ptr + 1] is guaranteed to be
+ * readable.
+ */
+ c2 = copy_input_buf[input_buf_ptr + 1];
+
+ simd_hit_eof = (c1 == '\\' && c2 == '.' && !is_csv);
+ simd_hit_eol = (c1 == '\r' || c1 == '\n');
+
+ /*
+ * If (is_csv && in_quote), we shouldn't have picked up '\r'
+ * or '\n' in the first place.
+ */
+ Assert(!simd_hit_eol || !(is_csv && in_quote));
+
+ /*
+ * Do not disable SIMD when we hit EOL or EOF characters. In
+ * practice, it does not matter for EOF because parsing ends
+ * there, but we keep the behavior consistent.
+ */
+ if (!(simd_hit_eof || simd_hit_eol))
+ {
+ simd_enabled = false;
+ cstate->simd_enabled = false;
+ }
+ }
+ else
+ {
+ /* No special characters found, so skip the entire chunk */
+ input_buf_ptr += sizeof(Vector8);
+ continue;
+ }
+ }
+#endif
+
/* OK to fetch a character */
prev_raw_ptr = input_buf_ptr;
c = copy_input_buf[input_buf_ptr++];
diff --git a/src/include/commands/copyfrom_internal.h b/src/include/commands/copyfrom_internal.h
index 822ef33cf69..73ce777c52b 100644
--- a/src/include/commands/copyfrom_internal.h
+++ b/src/include/commands/copyfrom_internal.h
@@ -89,6 +89,9 @@ typedef struct CopyFromStateData
const char *cur_attval; /* current att value for error messages */
bool relname_only; /* don't output line number, att, etc. */
+ /* SIMD variables */
+ bool simd_enabled;
+
/*
* Working state
*/
--
2.47.3
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
@ 2025-12-12 20:42 ` Mark Wong <markwkm@gmail.com>
2025-12-12 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
1 sibling, 1 reply; 178+ messages in thread
From: Mark Wong @ 2025-12-12 20:42 UTC (permalink / raw)
To: Bilal Yavuz <byavuz81@gmail.com>; +Cc: Manni Wood <manni.wood@enterprisedb.com>; KAZAR Ayoub <ma_kazar@esi.dz>; Nathan Bossart <nathandbossart@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
Hi everyone,
On Tue, Dec 09, 2025 at 04:40:19PM +0300, Bilal Yavuz wrote:
> Hi,
>
> On Sat, 6 Dec 2025 at 10:55, Bilal Yavuz <byavuz81@gmail.com> wrote:
> >
> > Hi,
> >
> > On Sat, 6 Dec 2025 at 04:40, Manni Wood <manni.wood@enterprisedb.com> wrote:
> > > Hello, all.
> > >
> > > Andrew, I tried your suggestion of just reading the first chunk of the copy file to determine if SIMD is worth using. Attached are v4 versions of the patches showing a first attempt at doing that.
> >
> > Thank you for doing this!
> >
> > > I attached test.sh.txt to show how I've been testing, with 5 million lines of the various copy file variations introduced by Ayub Kazar.
> > >
> > > The text copy with no special chars is 30% faster. The CSV copy with no special chars is 48% faster. The text with 1/3rd escapes is 3% slower. The CSV with 1/3rd quotes is 0.27% slower.
> > >
> > > This set of patches follows the simplest suggestion of just testing the first N lines (actually first N bytes) of the file and then deciding whether or not to enable SIMD. This set of patches does not follow Andrew's later suggestion of maybe checking again every million lines or so.
> >
> > My input-generation script is not ready to share yet, but the inputs
> > follow this format: text_${n}.input, where n represents the number of
> > normal characters before the delimiter. For example:
> >
> > n = 0 -> "\n\n\n\n\n..." (no normal characters)
> > n = 1 -> "a\n..." (1 normal character before the delimiter)
> > ...
> > n = 5 -> "aaaaa\n..."
> > … continuing up to n = 32.
> >
> > Each line has 4096 chars and there are a total of 100000 lines in each
> > input file.
> >
> > I only benchmarked the text format. I compared the latest heuristic I
> > shared [1] with the current method. The benchmarks show roughly a ~16%
> > regression at the worst case (n = 2), with regressions up to n = 5.
> > For the remaining values, performance was similar.
>
> I tried to improve the v4 patchset. My changes are:
>
> 1 - I changed CopyReadLineText() to an inline function and sent the
> use_simd variable as an argument to get help from inlining.
>
> 2 - A main for loop in the CopyReadLineText() function is called many
> times, so I moved the use_simd check to the CopyReadLine() function.
>
> 3 - Instead of 'bytes_processed', I used 'chars_processed' because
> cstate->bytes_processed is increased before we process them and this
> can cause wrong results.
>
> 4 - Because of #2 and #3, instead of having
> 'SPECIAL_CHAR_SIMD_THRESHOLD', I used the ratio of 'chars_processed /
> special_chars_encountered' to determine whether we want to use SIMD.
>
> 5 - cstate->special_chars_encountered is incremented wrongly for the
> CSV case. It is not incremented for the quote and escape delimiters. I
> moved all increments of cstate->special_chars_encountered to the
> central place and tried to optimize it but it still causes a
> regression as it creates one more branching.
>
> With these changes, I am able to decrease the regression to %10 from
> %16. Regression decreases to %7 if I modify #5 for the only text input
> but I did not do that.
>
> My changes are in the 0003.
I was helping collect some data, but I'm a little behind sharing what I
ran against the v4.1 patches (on commit 07961ef8) with the v4.2 version
out there...
I hope it's still helpfule that I share what I collected even though
they are not quite as nice, but maybe it's more about how/where I ran
them.
My laptop has a Intel(R) Core(TM) Ultra 7 165H, where most of these
tests were using up 95%+ of one of the cores (I have hyperthreading
disabled), and using about 10% the ssd's capacity.
Summarizing my results from the same script Manni ran, I didn't see as
much as an improvement in the positive tests, and then saw more negative
results in the other tests.
text copy with no special chars: 18% improvement of 15s from 80s before
the patch
CSV copy with no special chars: 23% improvement of 23s from 96s before
the patch
text with 1/3rd escapes: 6% slower, an additional 5s to 85 seconds
before the patch
CSV with 1/3rd quotes: 7% slower, an additional 10 seconds to 129
seconds before the patch
I'm wondering if my laptop/processor isn't the best test bed for this...
Regards,
Mark
--
Mark Wong <markwkm@gmail.com>
EDB https://enterprisedb.com
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-12 20:42 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Mark Wong <markwkm@gmail.com>
@ 2025-12-12 23:09 ` Manni Wood <manni.wood@enterprisedb.com>
2025-12-18 07:35 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
0 siblings, 1 reply; 178+ messages in thread
From: Manni Wood @ 2025-12-12 23:09 UTC (permalink / raw)
To: Mark Wong <markwkm@gmail.com>; +Cc: Bilal Yavuz <byavuz81@gmail.com>; KAZAR Ayoub <ma_kazar@esi.dz>; Nathan Bossart <nathandbossart@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
On Fri, Dec 12, 2025 at 2:42 PM Mark Wong <markwkm@gmail.com> wrote:
> Hi everyone,
>
> On Tue, Dec 09, 2025 at 04:40:19PM +0300, Bilal Yavuz wrote:
> > Hi,
> >
> > On Sat, 6 Dec 2025 at 10:55, Bilal Yavuz <byavuz81@gmail.com> wrote:
> > >
> > > Hi,
> > >
> > > On Sat, 6 Dec 2025 at 04:40, Manni Wood <manni.wood@enterprisedb.com>
> wrote:
> > > > Hello, all.
> > > >
> > > > Andrew, I tried your suggestion of just reading the first chunk of
> the copy file to determine if SIMD is worth using. Attached are v4 versions
> of the patches showing a first attempt at doing that.
> > >
> > > Thank you for doing this!
> > >
> > > > I attached test.sh.txt to show how I've been testing, with 5 million
> lines of the various copy file variations introduced by Ayub Kazar.
> > > >
> > > > The text copy with no special chars is 30% faster. The CSV copy with
> no special chars is 48% faster. The text with 1/3rd escapes is 3% slower.
> The CSV with 1/3rd quotes is 0.27% slower.
> > > >
> > > > This set of patches follows the simplest suggestion of just testing
> the first N lines (actually first N bytes) of the file and then deciding
> whether or not to enable SIMD. This set of patches does not follow Andrew's
> later suggestion of maybe checking again every million lines or so.
> > >
> > > My input-generation script is not ready to share yet, but the inputs
> > > follow this format: text_${n}.input, where n represents the number of
> > > normal characters before the delimiter. For example:
> > >
> > > n = 0 -> "\n\n\n\n\n..." (no normal characters)
> > > n = 1 -> "a\n..." (1 normal character before the delimiter)
> > > ...
> > > n = 5 -> "aaaaa\n..."
> > > … continuing up to n = 32.
> > >
> > > Each line has 4096 chars and there are a total of 100000 lines in each
> > > input file.
> > >
> > > I only benchmarked the text format. I compared the latest heuristic I
> > > shared [1] with the current method. The benchmarks show roughly a ~16%
> > > regression at the worst case (n = 2), with regressions up to n = 5.
> > > For the remaining values, performance was similar.
> >
> > I tried to improve the v4 patchset. My changes are:
> >
> > 1 - I changed CopyReadLineText() to an inline function and sent the
> > use_simd variable as an argument to get help from inlining.
> >
> > 2 - A main for loop in the CopyReadLineText() function is called many
> > times, so I moved the use_simd check to the CopyReadLine() function.
> >
> > 3 - Instead of 'bytes_processed', I used 'chars_processed' because
> > cstate->bytes_processed is increased before we process them and this
> > can cause wrong results.
> >
> > 4 - Because of #2 and #3, instead of having
> > 'SPECIAL_CHAR_SIMD_THRESHOLD', I used the ratio of 'chars_processed /
> > special_chars_encountered' to determine whether we want to use SIMD.
> >
> > 5 - cstate->special_chars_encountered is incremented wrongly for the
> > CSV case. It is not incremented for the quote and escape delimiters. I
> > moved all increments of cstate->special_chars_encountered to the
> > central place and tried to optimize it but it still causes a
> > regression as it creates one more branching.
> >
> > With these changes, I am able to decrease the regression to %10 from
> > %16. Regression decreases to %7 if I modify #5 for the only text input
> > but I did not do that.
> >
> > My changes are in the 0003.
>
> I was helping collect some data, but I'm a little behind sharing what I
> ran against the v4.1 patches (on commit 07961ef8) with the v4.2 version
> out there...
>
> I hope it's still helpfule that I share what I collected even though
> they are not quite as nice, but maybe it's more about how/where I ran
> them.
>
> My laptop has a Intel(R) Core(TM) Ultra 7 165H, where most of these
> tests were using up 95%+ of one of the cores (I have hyperthreading
> disabled), and using about 10% the ssd's capacity.
>
> Summarizing my results from the same script Manni ran, I didn't see as
> much as an improvement in the positive tests, and then saw more negative
> results in the other tests.
>
> text copy with no special chars: 18% improvement of 15s from 80s before
> the patch
>
> CSV copy with no special chars: 23% improvement of 23s from 96s before
> the patch
>
> text with 1/3rd escapes: 6% slower, an additional 5s to 85 seconds
> before the patch
>
> CSV with 1/3rd quotes: 7% slower, an additional 10 seconds to 129
> seconds before the patch
>
>
> I'm wondering if my laptop/processor isn't the best test bed for this...
>
> Regards,
> Mark
> --
> Mark Wong <markwkm@gmail.com>
> EDB https://enterprisedb.com
>
Hello, Everyone!
I have attached two files. 1) the shell script that Mark and I have been
using to get our test results, and 2) a screenshot of a spreadsheet of my
latest test results. (Please let me know if there's a different format than
a screenshot that I could share my spreadsheet in.)
I took greater care this time to compile all three variants of Postgres
(master at bfb335df, master at bfb335df with v4.2 patches installed, master
at bfb335df with v3 patches installed) with the same gcc optimization flags
that would be used to build Postgres packages. To the best of my knowledge,
the two gcc flags of greatest interest would be -g and -O2. I built all
three variants of Postgres using meson like so:
BRANCH=$(git branch --show-current)
meson setup build --prefix=/home/mwood/compiled-pg-instances/${BRANCH}
--buildtype=debugoptimized
It occurred to me that in addition to end users only caring about 1) wall
clock time (is the speedup noticeable in "real time" or just technically
faster / uses less CPU?) and 2) Postgres binaries compiled with the same
optimization level one would get when installing Postgres from packages
like .deb or .rpm; in other words, will the user see speedups without
having do manually compile postgres.
My interesting finding, on my laptop (ThinkPad P14s Gen 1 running Ubuntu
24.04.3), is different from Mark Wong's. On my laptop, using three Postgres
installations all compiled with the -O2 optimization flag, I see speedups
with the v4.2 patch except for a 2% slowdown with CSV with 1/3rd quotes (a
2% slowdown). But with Nazir's proposed v3 patch, I see improvements across
the board. So even for a text file with 1/3rd escape characters, and even
with a CSV file with 1/3rd quotes, I see speedups of 11% and 26%
respectively.
The format of these test files originally comes from Ayoub Kazar's test
scripts; all Mark and I have done in playing with them is make them much
larger: 5,000,000 rows, based on the assumption that longer tests are
better tests.
I find my results interesting enough that I'd be curious to know if anybody
else can reproduce them. It is very interesting that Mark's results are
noticeably different from mine.
--
-- Manni Wood EDB: https://www.enterprisedb.com
Attachments:
[application/x-shellscript] manni-simd-copy-bench-v1.2.1.sh (2.6K, ../../CAKWEB6r=axZsG-s7zyWURZ-s9-s1dTV9ohkZXO0ynfLEU5ha3Q@mail.gmail.com/3-manni-simd-copy-bench-v1.2.1.sh)
download
[image/png] simd_copy_performance_2025_12_12.png (364.8K, ../../CAKWEB6r=axZsG-s7zyWURZ-s9-s1dTV9ohkZXO0ynfLEU5ha3Q@mail.gmail.com/4-simd_copy_performance_2025_12_12.png)
download | view image
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-12 20:42 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Mark Wong <markwkm@gmail.com>
2025-12-12 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
@ 2025-12-18 07:35 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-12-24 15:07 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
0 siblings, 1 reply; 178+ messages in thread
From: Nazir Bilal Yavuz @ 2025-12-18 07:35 UTC (permalink / raw)
To: Manni Wood <manni.wood@enterprisedb.com>; +Cc: Mark Wong <markwkm@gmail.com>; KAZAR Ayoub <ma_kazar@esi.dz>; Nathan Bossart <nathandbossart@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
Hi,
On Sat, 13 Dec 2025 at 02:09, Manni Wood <manni.wood@enterprisedb.com> wrote:
>
> Hello, Everyone!
>
> I have attached two files. 1) the shell script that Mark and I have been using to get our test results, and 2) a screenshot of a spreadsheet of my latest test results. (Please let me know if there's a different format than a screenshot that I could share my spreadsheet in.)
>
> I took greater care this time to compile all three variants of Postgres (master at bfb335df, master at bfb335df with v4.2 patches installed, master at bfb335df with v3 patches installed) with the same gcc optimization flags that would be used to build Postgres packages. To the best of my knowledge, the two gcc flags of greatest interest would be -g and -O2. I built all three variants of Postgres using meson like so:
>
> BRANCH=$(git branch --show-current)
> meson setup build --prefix=/home/mwood/compiled-pg-instances/${BRANCH} --buildtype=debugoptimized
>
> It occurred to me that in addition to end users only caring about 1) wall clock time (is the speedup noticeable in "real time" or just technically faster / uses less CPU?) and 2) Postgres binaries compiled with the same optimization level one would get when installing Postgres from packages like .deb or .rpm; in other words, will the user see speedups without having do manually compile postgres.
>
> My interesting finding, on my laptop (ThinkPad P14s Gen 1 running Ubuntu 24.04.3), is different from Mark Wong's. On my laptop, using three Postgres installations all compiled with the -O2 optimization flag, I see speedups with the v4.2 patch except for a 2% slowdown with CSV with 1/3rd quotes (a 2% slowdown). But with Nazir's proposed v3 patch, I see improvements across the board. So even for a text file with 1/3rd escape characters, and even with a CSV file with 1/3rd quotes, I see speedups of 11% and 26% respectively.
>
> The format of these test files originally comes from Ayoub Kazar's test scripts; all Mark and I have done in playing with them is make them much larger: 5,000,000 rows, based on the assumption that longer tests are better tests.
>
> I find my results interesting enough that I'd be curious to know if anybody else can reproduce them. It is very interesting that Mark's results are noticeably different from mine.
Thank you for sharing the benchmark script! I ran the benchmarks using
your script with --buildtype=debugoptimized. My results are below:
master: 85ddcc2f4c
text, no special: 102294
text, 1/3 special: 108946
csv, no special: 121831
csv, 1/3 special: 140063
v3
text, no special: 88890 (13.1% speedup)
text, 1/3 special: 110463 (1.4% regression)
csv, no special: 89781 (26.3% speedup)
csv, 1/3 special: 147094 (5.0% regression)
v4.2
text, no special: 87785 (14.2% speedup)
text, 1/3 special: 127008 (16.6% regression)
csv, no special: 88093 (27.7% speedup)
csv, 1/3 special: 164487 (17.4% regression)
One thing I noticed is that your benchmark timings appear to have some
variance. In my runs, I did not observe differences greater than one
second between runs. It is possible that this variance is affecting
your results.
Before running the benchmarks, I use the these commands [1] to improve
result stability; they might be helpful if you are not already using
something similar:
I did this benchmark on my local and my specs are Intel i5 13600k,
32GB Memory and SATA SSD.
[1]
sudo cpupower frequency-set --governor=performance
sudo cpupower idle-set -D 0 # disable idle
echo "1" | sudo tee /sys/devices/system/cpu/intel_pstate/no_turbo (intel only)
--
Regards,
Nazir Bilal Yavuz
Microsoft
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-12 20:42 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Mark Wong <markwkm@gmail.com>
2025-12-12 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-18 07:35 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
@ 2025-12-24 15:07 ` KAZAR Ayoub <ma_kazar@esi.dz>
2025-12-29 17:03 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-31 13:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
0 siblings, 2 replies; 178+ messages in thread
From: KAZAR Ayoub @ 2025-12-24 15:07 UTC (permalink / raw)
To: Nazir Bilal Yavuz <byavuz81@gmail.com>; +Cc: Manni Wood <manni.wood@enterprisedb.com>; Mark Wong <markwkm@gmail.com>; Nathan Bossart <nathandbossart@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
Hello,
Following the same path of optimizing COPY FROM using SIMD, i found that
COPY TO can also benefit from this.
I attached a small patch that uses SIMD to skip data and advance as far as
the first special character is found, then fallback to scalar processing
for that character and re-enter the SIMD path again...
There's two ways to do this:
1) Essentially we do SIMD until we find a special character, then continue
scalar path without re-entering SIMD again.
- This gives from 10% to 30% speedups depending on the weight of special
characters in the attribute, we don't lose anything here since it advances
with SIMD until it can't (using the previous scripts: 1/3, 2/3 specials
chars).
2) Do SIMD path, then use scalar path when we hit a special character, keep
re-entering the SIMD path each time.
- This is equivalent to the COPY FROM story, we'll need to find the same
heuristic to use for both COPY FROM/TO to reduce the regressions (same
regressions: around from 20% to 30% with 1/3, 2/3 specials chars).
Something else to note is that the scalar path for COPY TO isn't as heavy
as the state machine in COPY FROM.
So if we find the sweet spot for the heuristic, doing the same for COPY TO
will be trivial and always beneficial.
Attached is 0004 which is option 1 (SIMD without re-entering), 0005 is the
second one.
Regards,
Ayoub
Attachments:
[text/x-patch] 0005-Speed-up-COPY-TO-text-CSV-using-SIMD.patch (9.2K, ../../CA+K2RumOaH-daBGN6uTo6+_0XSg7HQ10Na8OzScCV5j6eKkFgA@mail.gmail.com/3-0005-Speed-up-COPY-TO-text-CSV-using-SIMD.patch)
download | inline diff:
From 319e5402e35429943d80ba136f27e6185410e6f5 Mon Sep 17 00:00:00 2001
From: AyoubKAZ <kazarayoub2004@gmail.com>
Date: Wed, 24 Dec 2025 15:20:53 +0100
Subject: [PATCH] Speed up COPY TO text CSV using SIMD
---
src/backend/commands/copyto.c | 252 ++++++++++++++++++++++------------
1 file changed, 167 insertions(+), 85 deletions(-)
diff --git a/src/backend/commands/copyto.c b/src/backend/commands/copyto.c
index e1306728509..b9d7b55f1ab 100644
--- a/src/backend/commands/copyto.c
+++ b/src/backend/commands/copyto.c
@@ -1268,38 +1268,63 @@ CopyAttributeOutText(CopyToState cstate, const char *string)
if (cstate->encoding_embeds_ascii)
{
start = ptr;
- #ifndef USE_NO_SIMD
+ const char *end = ptr + strlen(ptr);
+
+ while ((c = *ptr) != '\0')
+ {
+#ifndef USE_NO_SIMD
+ /*
+ * SIMD fast path: scan ahead for special characters.
+ * We re-enter this path after handling each special character
+ * to maximize the benefit of vectorization.
+ */
{
- const char* end = ptr + strlen(ptr);
- while (ptr + sizeof(Vector8) <= end) {
- Vector8 chunk;
- Vector8 control_mask;
- Vector8 backslash_mask;
- Vector8 delim_mask;
- Vector8 special_mask;
- uint32 mask;
+
+ while (ptr + sizeof(Vector8) <= end)
+ {
+ Vector8 chunk;
+ Vector8 control_mask;
+ Vector8 backslash_mask;
+ Vector8 delim_mask;
+ Vector8 special_mask;
+ uint32 mask;
vector8_load(&chunk, (const uint8 *) ptr);
+
+ /* Check for control characters (< 0x20) */
control_mask = vector8_gt(vector8_broadcast(0x20), chunk);
- backslash_mask = vector8_eq(vector8_broadcast('\\'), chunk);
- delim_mask = vector8_eq(vector8_broadcast(delimc), chunk);
+
+ /* Check for backslash and delimiter */
+ backslash_mask = vector8_eq(chunk, vector8_broadcast('\\'));
+ delim_mask = vector8_eq(chunk, vector8_broadcast(delimc));
+
- special_mask = vector8_or(control_mask, vector8_or(backslash_mask, delim_mask));
+ /* Combine all masks */
+ special_mask = vector8_or(
+ vector8_or(control_mask, backslash_mask), delim_mask);
mask = vector8_highbit_mask(special_mask);
- if (mask != 0) {
+ if (mask != 0)
+ {
+ /* Found special character, advance to it */
int advance = pg_rightmost_one_pos32(mask);
ptr += advance;
break;
}
+ /* No special characters in this chunk, advance */
ptr += sizeof(Vector8);
}
- }
- #endif
+
+ /* Update c after SIMD scan */
+ c = *ptr;
+ }
+#endif /* !USE_NO_SIMD */
+
+ /* Scalar handling - same code for SIMD and non-SIMD builds */
+ if (c == '\0')
+ break;
- while ((c = *ptr) != '\0')
- {
if ((unsigned char) c < (unsigned char) 0x20)
{
/*
@@ -1358,38 +1383,60 @@ CopyAttributeOutText(CopyToState cstate, const char *string)
else
{
start = ptr;
- #ifndef USE_NO_SIMD
+ const char *end = ptr + strlen(ptr);
+
+ while ((c = *ptr) != '\0')
+ {
+#ifndef USE_NO_SIMD
+ /*
+ * SIMD fast path: scan ahead for special characters.
+ */
{
- const char* end = ptr + strlen(ptr);
- while (ptr + sizeof(Vector8) <= end) {
- Vector8 chunk;
- Vector8 control_mask;
- Vector8 backslash_mask;
- Vector8 delim_mask;
- Vector8 special_mask;
- uint32 mask;
+
+ while (ptr + sizeof(Vector8) <= end)
+ {
+ Vector8 chunk;
+ Vector8 control_mask;
+ Vector8 backslash_mask;
+ Vector8 delim_mask;
+ Vector8 special_mask;
+ uint32 mask;
vector8_load(&chunk, (const uint8 *) ptr);
+
+ /* Check for control characters (< 0x20) */
control_mask = vector8_gt(vector8_broadcast(0x20), chunk);
- backslash_mask = vector8_eq(vector8_broadcast('\\'), chunk);
- delim_mask = vector8_eq(vector8_broadcast(delimc), chunk);
+
+ /* Check for backslash and delimiter */
+ backslash_mask = vector8_eq(chunk, vector8_broadcast('\\'));
+ delim_mask = vector8_eq(chunk, vector8_broadcast(delimc));
- special_mask = vector8_or(control_mask, vector8_or(backslash_mask, delim_mask));
+ /* Combine masks */
+ special_mask = vector8_or(control_mask,
+ vector8_or(backslash_mask, delim_mask));
mask = vector8_highbit_mask(special_mask);
- if (mask != 0) {
+ if (mask != 0)
+ {
+ /* Found special character */
int advance = pg_rightmost_one_pos32(mask);
ptr += advance;
break;
}
+ /* No special characters, advance */
ptr += sizeof(Vector8);
}
- }
- #endif
+
+ /* Update c after SIMD scan */
+ c = *ptr;
+ }
+#endif /* !USE_NO_SIMD */
+
+ /* Scalar handling - same for SIMD and non-SIMD */
+ if (c == '\0')
+ break;
- while ((c = *ptr) != '\0')
- {
if ((unsigned char) c < (unsigned char) 0x20)
{
/*
@@ -1489,53 +1536,68 @@ CopyAttributeOutCSV(CopyToState cstate, const char *string,
else
{
const char *tptr = ptr;
+ const char *end = tptr + strlen(tptr);
+
+ while ((c = *tptr) != '\0')
+ {
+#ifndef USE_NO_SIMD
+ /*
+ * SIMD accelerated quote detection.
+ */
+ {
+ Vector8 delim_vec;
+ Vector8 quote_vec;
+ Vector8 newline_vec;
+ Vector8 cr_vec;
+
+ delim_vec = vector8_broadcast(delimc);
+ quote_vec = vector8_broadcast(quotec);
+ newline_vec = vector8_broadcast('\n');
+ cr_vec = vector8_broadcast('\r');
+
+ while (tptr + sizeof(Vector8) <= end)
+ {
+ Vector8 chunk;
+ Vector8 special_mask;
+ uint32 mask;
- #ifndef USE_NO_SIMD
- {
- const char* end = tptr + strlen(tptr);
-
- Vector8 delim_mask = vector8_broadcast(delimc);
- Vector8 quote_mask = vector8_broadcast(quotec);
- Vector8 newline_mask = vector8_broadcast('\n');
- Vector8 carriage_return_mask = vector8_broadcast('\r');
-
- while (tptr + sizeof(Vector8) <= end) {
- Vector8 chunk;
- Vector8 special_mask;
- uint32 mask;
-
- vector8_load(&chunk, (const uint8 *) tptr);
- special_mask = vector8_or(
- vector8_or(vector8_eq(chunk, delim_mask),
- vector8_eq(chunk, quote_mask)),
- vector8_or(vector8_eq(chunk, newline_mask),
- vector8_eq(chunk, carriage_return_mask))
- );
-
- mask = vector8_highbit_mask(special_mask);
- if (mask != 0) {
- tptr += pg_rightmost_one_pos32(mask);
- use_quote = true;
- break;
- }
+ vector8_load(&chunk, (const uint8 *) tptr);
+
+ special_mask = vector8_or(
+ vector8_or(vector8_eq(chunk, delim_vec),
+ vector8_eq(chunk, quote_vec)),
+ vector8_or(vector8_eq(chunk, newline_vec),
+ vector8_eq(chunk, cr_vec)));
- tptr += sizeof(Vector8);
+ mask = vector8_highbit_mask(special_mask);
+ if (mask != 0)
+ {
+ tptr += pg_rightmost_one_pos32(mask);
+ use_quote = true;
+ break;
}
+
+ tptr += sizeof(Vector8);
}
- #endif
+ }
+#endif /* !USE_NO_SIMD */
- while ((c = *tptr) != '\0')
+ /*
+ * Scalar scan for remaining bytes (tail after SIMD, or entire
+ * string if USE_NO_SIMD).
+ */
+ if ((c = *tptr) != '\0')
{
if (c == delimc || c == quotec || c == '\n' || c == '\r')
{
use_quote = true;
- break;
}
if (IS_HIGHBIT_SET(c) && cstate->encoding_embeds_ascii)
tptr += pg_encoding_mblen(cstate->file_encoding, tptr);
else
tptr++;
}
+ }
}
}
@@ -1548,37 +1610,57 @@ CopyAttributeOutCSV(CopyToState cstate, const char *string,
*/
start = ptr;
- #ifndef USE_NO_SIMD
- {
- const char* end = ptr + strlen(ptr);
-
- Vector8 escape_mask = vector8_broadcast(escapec);
- Vector8 quote_mask = vector8_broadcast(quotec);
+ const char *end = ptr + strlen(ptr);
- while (ptr + sizeof(Vector8) <= end) {
- Vector8 chunk;
- Vector8 special_mask;
- uint32 mask;
+ while ((c = *ptr) != '\0')
+ {
+#ifndef USE_NO_SIMD
+ /*
+ * SIMD fast path: scan ahead for quote/escape characters.
+ * Re-enter after handling each special character.
+ */
+ {
+ Vector8 escape_vec;
+ Vector8 quote_vec;
+
+ /* Pre-compute broadcast vectors */
+ escape_vec = vector8_broadcast(escapec);
+ quote_vec = vector8_broadcast(quotec);
+
+ while (ptr + sizeof(Vector8) <= end)
+ {
+ Vector8 chunk;
+ Vector8 special_mask;
+ uint32 mask;
vector8_load(&chunk, (const uint8 *) ptr);
+
special_mask = vector8_or(
- vector8_eq(chunk, escape_mask),
- vector8_eq(chunk, quote_mask));
+ vector8_eq(chunk, escape_vec),
+ vector8_eq(chunk, quote_vec));
mask = vector8_highbit_mask(special_mask);
- if (mask != 0) {
- ptr += pg_rightmost_one_pos32(mask);
- use_quote = true;
+ if (mask != 0)
+ {
+ /* Found special character */
+ int advance = pg_rightmost_one_pos32(mask);
+ ptr += advance;
break;
}
+ /* No special characters in this chunk */
ptr += sizeof(Vector8);
}
+
+ /* Update c after SIMD scan */
+ c = *ptr;
}
- #endif
-
- while ((c = *ptr) != '\0')
- {
+#endif /* !USE_NO_SIMD */
+
+ /* Scalar handling - same code for SIMD and non-SIMD builds */
+ if (c == '\0')
+ break;
+
if (c == quotec || c == escapec)
{
DUMPSOFAR();
--
2.34.1
[text/x-patch] 0004-Speed-up-COPY-TO-text-CSV-using-SIMD.patch (4.8K, ../../CA+K2RumOaH-daBGN6uTo6+_0XSg7HQ10Na8OzScCV5j6eKkFgA@mail.gmail.com/4-0004-Speed-up-COPY-TO-text-CSV-using-SIMD.patch)
download | inline diff:
From bfc580b17ad5e6d981adc146c24690afe4634ce1 Mon Sep 17 00:00:00 2001
From: AyoubKAZ <kazarayoub2004@gmail.com>
Date: Wed, 24 Dec 2025 12:55:15 +0100
Subject: [PATCH] Speed up COPY TO text CSV using SIMD
---
src/backend/commands/copyto.c | 126 ++++++++++++++++++++++++++++++++++
1 file changed, 126 insertions(+)
diff --git a/src/backend/commands/copyto.c b/src/backend/commands/copyto.c
index dae91630ac3..e1306728509 100644
--- a/src/backend/commands/copyto.c
+++ b/src/backend/commands/copyto.c
@@ -31,6 +31,8 @@
#include "mb/pg_wchar.h"
#include "miscadmin.h"
#include "pgstat.h"
+#include "port/pg_bitutils.h"
+#include "port/simd.h"
#include "storage/fd.h"
#include "tcop/tcopprot.h"
#include "utils/lsyscache.h"
@@ -1266,6 +1268,36 @@ CopyAttributeOutText(CopyToState cstate, const char *string)
if (cstate->encoding_embeds_ascii)
{
start = ptr;
+ #ifndef USE_NO_SIMD
+ {
+ const char* end = ptr + strlen(ptr);
+ while (ptr + sizeof(Vector8) <= end) {
+ Vector8 chunk;
+ Vector8 control_mask;
+ Vector8 backslash_mask;
+ Vector8 delim_mask;
+ Vector8 special_mask;
+ uint32 mask;
+
+ vector8_load(&chunk, (const uint8 *) ptr);
+ control_mask = vector8_gt(vector8_broadcast(0x20), chunk);
+ backslash_mask = vector8_eq(vector8_broadcast('\\'), chunk);
+ delim_mask = vector8_eq(vector8_broadcast(delimc), chunk);
+
+ special_mask = vector8_or(control_mask, vector8_or(backslash_mask, delim_mask));
+
+ mask = vector8_highbit_mask(special_mask);
+ if (mask != 0) {
+ int advance = pg_rightmost_one_pos32(mask);
+ ptr += advance;
+ break;
+ }
+
+ ptr += sizeof(Vector8);
+ }
+ }
+ #endif
+
while ((c = *ptr) != '\0')
{
if ((unsigned char) c < (unsigned char) 0x20)
@@ -1326,6 +1358,36 @@ CopyAttributeOutText(CopyToState cstate, const char *string)
else
{
start = ptr;
+ #ifndef USE_NO_SIMD
+ {
+ const char* end = ptr + strlen(ptr);
+ while (ptr + sizeof(Vector8) <= end) {
+ Vector8 chunk;
+ Vector8 control_mask;
+ Vector8 backslash_mask;
+ Vector8 delim_mask;
+ Vector8 special_mask;
+ uint32 mask;
+
+ vector8_load(&chunk, (const uint8 *) ptr);
+ control_mask = vector8_gt(vector8_broadcast(0x20), chunk);
+ backslash_mask = vector8_eq(vector8_broadcast('\\'), chunk);
+ delim_mask = vector8_eq(vector8_broadcast(delimc), chunk);
+
+ special_mask = vector8_or(control_mask, vector8_or(backslash_mask, delim_mask));
+
+ mask = vector8_highbit_mask(special_mask);
+ if (mask != 0) {
+ int advance = pg_rightmost_one_pos32(mask);
+ ptr += advance;
+ break;
+ }
+
+ ptr += sizeof(Vector8);
+ }
+ }
+ #endif
+
while ((c = *ptr) != '\0')
{
if ((unsigned char) c < (unsigned char) 0x20)
@@ -1428,6 +1490,40 @@ CopyAttributeOutCSV(CopyToState cstate, const char *string,
{
const char *tptr = ptr;
+ #ifndef USE_NO_SIMD
+ {
+ const char* end = tptr + strlen(tptr);
+
+ Vector8 delim_mask = vector8_broadcast(delimc);
+ Vector8 quote_mask = vector8_broadcast(quotec);
+ Vector8 newline_mask = vector8_broadcast('\n');
+ Vector8 carriage_return_mask = vector8_broadcast('\r');
+
+ while (tptr + sizeof(Vector8) <= end) {
+ Vector8 chunk;
+ Vector8 special_mask;
+ uint32 mask;
+
+ vector8_load(&chunk, (const uint8 *) tptr);
+ special_mask = vector8_or(
+ vector8_or(vector8_eq(chunk, delim_mask),
+ vector8_eq(chunk, quote_mask)),
+ vector8_or(vector8_eq(chunk, newline_mask),
+ vector8_eq(chunk, carriage_return_mask))
+ );
+
+ mask = vector8_highbit_mask(special_mask);
+ if (mask != 0) {
+ tptr += pg_rightmost_one_pos32(mask);
+ use_quote = true;
+ break;
+ }
+
+ tptr += sizeof(Vector8);
+ }
+ }
+ #endif
+
while ((c = *tptr) != '\0')
{
if (c == delimc || c == quotec || c == '\n' || c == '\r')
@@ -1451,6 +1547,36 @@ CopyAttributeOutCSV(CopyToState cstate, const char *string,
* We adopt the same optimization strategy as in CopyAttributeOutText
*/
start = ptr;
+
+ #ifndef USE_NO_SIMD
+ {
+ const char* end = ptr + strlen(ptr);
+
+ Vector8 escape_mask = vector8_broadcast(escapec);
+ Vector8 quote_mask = vector8_broadcast(quotec);
+
+ while (ptr + sizeof(Vector8) <= end) {
+ Vector8 chunk;
+ Vector8 special_mask;
+ uint32 mask;
+
+ vector8_load(&chunk, (const uint8 *) ptr);
+ special_mask = vector8_or(
+ vector8_eq(chunk, escape_mask),
+ vector8_eq(chunk, quote_mask));
+
+ mask = vector8_highbit_mask(special_mask);
+ if (mask != 0) {
+ ptr += pg_rightmost_one_pos32(mask);
+ use_quote = true;
+ break;
+ }
+
+ ptr += sizeof(Vector8);
+ }
+ }
+ #endif
+
while ((c = *ptr) != '\0')
{
if (c == quotec || c == escapec)
--
2.34.1
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-12 20:42 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Mark Wong <markwkm@gmail.com>
2025-12-12 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-18 07:35 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-12-24 15:07 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
@ 2025-12-29 17:03 ` Manni Wood <manni.wood@enterprisedb.com>
1 sibling, 0 replies; 178+ messages in thread
From: Manni Wood @ 2025-12-29 17:03 UTC (permalink / raw)
To: KAZAR Ayoub <ma_kazar@esi.dz>; +Cc: Nazir Bilal Yavuz <byavuz81@gmail.com>; Mark Wong <markwkm@gmail.com>; Nathan Bossart <nathandbossart@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
On Wed, Dec 24, 2025 at 9:08 AM KAZAR Ayoub <ma_kazar@esi.dz> wrote:
> Hello,
> Following the same path of optimizing COPY FROM using SIMD, i found that
> COPY TO can also benefit from this.
>
> I attached a small patch that uses SIMD to skip data and advance as far as
> the first special character is found, then fallback to scalar processing
> for that character and re-enter the SIMD path again...
> There's two ways to do this:
> 1) Essentially we do SIMD until we find a special character, then continue
> scalar path without re-entering SIMD again.
> - This gives from 10% to 30% speedups depending on the weight of special
> characters in the attribute, we don't lose anything here since it advances
> with SIMD until it can't (using the previous scripts: 1/3, 2/3 specials
> chars).
>
> 2) Do SIMD path, then use scalar path when we hit a special character,
> keep re-entering the SIMD path each time.
> - This is equivalent to the COPY FROM story, we'll need to find the same
> heuristic to use for both COPY FROM/TO to reduce the regressions (same
> regressions: around from 20% to 30% with 1/3, 2/3 specials chars).
>
> Something else to note is that the scalar path for COPY TO isn't as heavy
> as the state machine in COPY FROM.
>
> So if we find the sweet spot for the heuristic, doing the same for COPY TO
> will be trivial and always beneficial.
> Attached is 0004 which is option 1 (SIMD without re-entering), 0005 is the
> second one.
>
>
> Regards,
> Ayoub
>
Hello, Nazir and Ayoub!
Nazir, sorry for the late reply, I am on holiday. :-) I wanted to thank you
for the tips on using cpupower to get less variance in my test results.
Ayoub, I suppose it was inevitable the SIMD patch would work for copying
out as well as copying in!
I am back at work on 5 Jan 2026, so I till try to carve out time to test
this then, using Nazir's tips.
Happy Holidays!
-Manni
--
-- Manni Wood EDB: https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-12 20:42 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Mark Wong <markwkm@gmail.com>
2025-12-12 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-18 07:35 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-12-24 15:07 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
@ 2025-12-31 13:04 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-06 20:05 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-01-17 20:46 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
1 sibling, 2 replies; 178+ messages in thread
From: Nazir Bilal Yavuz @ 2025-12-31 13:04 UTC (permalink / raw)
To: KAZAR Ayoub <ma_kazar@esi.dz>; +Cc: Manni Wood <manni.wood@enterprisedb.com>; Mark Wong <markwkm@gmail.com>; Nathan Bossart <nathandbossart@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
Hi,
On Wed, 24 Dec 2025 at 18:08, KAZAR Ayoub <ma_kazar@esi.dz> wrote:
>
> Hello,
> Following the same path of optimizing COPY FROM using SIMD, i found that COPY TO can also benefit from this.
>
> I attached a small patch that uses SIMD to skip data and advance as far as the first special character is found, then fallback to scalar processing for that character and re-enter the SIMD path again...
> There's two ways to do this:
> 1) Essentially we do SIMD until we find a special character, then continue scalar path without re-entering SIMD again.
> - This gives from 10% to 30% speedups depending on the weight of special characters in the attribute, we don't lose anything here since it advances with SIMD until it can't (using the previous scripts: 1/3, 2/3 specials chars).
>
> 2) Do SIMD path, then use scalar path when we hit a special character, keep re-entering the SIMD path each time.
> - This is equivalent to the COPY FROM story, we'll need to find the same heuristic to use for both COPY FROM/TO to reduce the regressions (same regressions: around from 20% to 30% with 1/3, 2/3 specials chars).
>
> Something else to note is that the scalar path for COPY TO isn't as heavy as the state machine in COPY FROM.
>
> So if we find the sweet spot for the heuristic, doing the same for COPY TO will be trivial and always beneficial.
> Attached is 0004 which is option 1 (SIMD without re-entering), 0005 is the second one.
Patches look correct to me. I think we could move these SIMD code
portions into a shared function to remove duplication, although that
might have a performance impact. I have not benchmarked these patches
yet.
Another consideration is that these patches might need their own
thread, though I am not completely sure about this yet.
One question: what do you think about having a 0004-style approach for
COPY FROM? What I have in mind is running SIMD for each line & column,
stopping SIMD once it can no longer skip an entire chunk, and then
continuing with the next line & column.
--
Regards,
Nazir Bilal Yavuz
Microsoft
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-12 20:42 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Mark Wong <markwkm@gmail.com>
2025-12-12 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-18 07:35 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-12-24 15:07 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-12-31 13:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
@ 2026-01-06 20:05 ` Manni Wood <manni.wood@enterprisedb.com>
2026-01-07 19:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-01-09 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
1 sibling, 2 replies; 178+ messages in thread
From: Manni Wood @ 2026-01-06 20:05 UTC (permalink / raw)
To: Nazir Bilal Yavuz <byavuz81@gmail.com>; +Cc: KAZAR Ayoub <ma_kazar@esi.dz>; Mark Wong <markwkm@gmail.com>; Nathan Bossart <nathandbossart@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
On Wed, Dec 31, 2025 at 7:04 AM Nazir Bilal Yavuz <byavuz81@gmail.com>
wrote:
> Hi,
>
> On Wed, 24 Dec 2025 at 18:08, KAZAR Ayoub <ma_kazar@esi.dz> wrote:
> >
> > Hello,
> > Following the same path of optimizing COPY FROM using SIMD, i found that
> COPY TO can also benefit from this.
> >
> > I attached a small patch that uses SIMD to skip data and advance as far
> as the first special character is found, then fallback to scalar processing
> for that character and re-enter the SIMD path again...
> > There's two ways to do this:
> > 1) Essentially we do SIMD until we find a special character, then
> continue scalar path without re-entering SIMD again.
> > - This gives from 10% to 30% speedups depending on the weight of special
> characters in the attribute, we don't lose anything here since it advances
> with SIMD until it can't (using the previous scripts: 1/3, 2/3 specials
> chars).
> >
> > 2) Do SIMD path, then use scalar path when we hit a special character,
> keep re-entering the SIMD path each time.
> > - This is equivalent to the COPY FROM story, we'll need to find the same
> heuristic to use for both COPY FROM/TO to reduce the regressions (same
> regressions: around from 20% to 30% with 1/3, 2/3 specials chars).
> >
> > Something else to note is that the scalar path for COPY TO isn't as
> heavy as the state machine in COPY FROM.
> >
> > So if we find the sweet spot for the heuristic, doing the same for COPY
> TO will be trivial and always beneficial.
> > Attached is 0004 which is option 1 (SIMD without re-entering), 0005 is
> the second one.
>
> Patches look correct to me. I think we could move these SIMD code
> portions into a shared function to remove duplication, although that
> might have a performance impact. I have not benchmarked these patches
> yet.
>
> Another consideration is that these patches might need their own
> thread, though I am not completely sure about this yet.
>
> One question: what do you think about having a 0004-style approach for
> COPY FROM? What I have in mind is running SIMD for each line & column,
> stopping SIMD once it can no longer skip an entire chunk, and then
> continuing with the next line & column.
>
> --
> Regards,
> Nazir Bilal Yavuz
> Microsoft
>
Hello, Nazir, I tried your suggested cpupower commands as well as disabling
turbo, and my results are indeed more uniform. (see attached screenshot of
my spreadsheet).
This time, I ran the tests on my Tower PC instead of on my laptop.
I also followed Mark Wong's advice and used the taskset command to pin my
postgres postmaster (and all of its children) to a single cpu core.
So when I start postgres, I do this to pin it to core 27:
${PGHOME}/bin/pg_ctl -D ${PGHOME}/data -l ${PGHOME}/logfile.txt start
PGPID=$(head -1 ${PGHOME}/data/postmaster.pid)
taskset --cpu-list -p 27 ${PGPID}
My results seem similar to yours:
master: Nazir 85ddcc2f4c | Manni 877ae5db
text, no special: 102294 | 302651
text, 1/3 special: 108946 | 326208
csv, no special: 121831 | 348930
csv, 1/3 special: 140063 | 439786
v3
text, no special: 88890 (13.1% speedup) | 227874 (24.7% speedup)
text, 1/3 special: 110463 (1.4% regression) | 322637 (1.1% speedup)
csv, no special: 89781 (26.3% speedup) | 226525 (35.1% speedup)
csv, 1/3 special: 147094 (5.0% regression) | 461501 (4.9% regression)
v4.2
text, no special: 87785 (14.2% speedup) | 225702 (25.4% speedup)
text, 1/3 special: 127008 (16.6% regression) | 343480 (5.3% regression)
csv, no special: 88093 (27.7% speedup) | 226633 (35.0% speedup)
csv, 1/3 special: 164487 (17.4% regression) | 510954 (16.2% regression)
It would seem that both your results and mine show a more serious
worst-case regression for the v4.2 patches than for the v3 patches. It
seems also that the speedups for v4.2 and v3 are similar.
I'm currently working with Mark Wong to see if his results continue to be
dissimilar (as they currently are now) and, if so, why.
--
-- Manni Wood EDB: https://www.enterprisedb.com
Attachments:
[image/png] simd_copy_performance_2025_01_06.png (114.9K, ../../CAKWEB6qQmPejLwndzMcVqhO+u9vgUf44kYAKg4QtyG-=KH=YGg@mail.gmail.com/3-simd_copy_performance_2025_01_06.png)
download | view image
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-12 20:42 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Mark Wong <markwkm@gmail.com>
2025-12-12 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-18 07:35 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-12-24 15:07 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-12-31 13:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-06 20:05 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
@ 2026-01-07 19:13 ` Manni Wood <manni.wood@enterprisedb.com>
2026-01-08 20:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-01-09 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
1 sibling, 2 replies; 178+ messages in thread
From: Manni Wood @ 2026-01-07 19:13 UTC (permalink / raw)
To: Nazir Bilal Yavuz <byavuz81@gmail.com>; +Cc: KAZAR Ayoub <ma_kazar@esi.dz>; Mark Wong <markwkm@gmail.com>; Nathan Bossart <nathandbossart@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
On Tue, Jan 6, 2026 at 2:05 PM Manni Wood <manni.wood@enterprisedb.com>
wrote:
>
>
> On Wed, Dec 31, 2025 at 7:04 AM Nazir Bilal Yavuz <byavuz81@gmail.com>
> wrote:
>
>> Hi,
>>
>> On Wed, 24 Dec 2025 at 18:08, KAZAR Ayoub <ma_kazar@esi.dz> wrote:
>> >
>> > Hello,
>> > Following the same path of optimizing COPY FROM using SIMD, i found
>> that COPY TO can also benefit from this.
>> >
>> > I attached a small patch that uses SIMD to skip data and advance as far
>> as the first special character is found, then fallback to scalar processing
>> for that character and re-enter the SIMD path again...
>> > There's two ways to do this:
>> > 1) Essentially we do SIMD until we find a special character, then
>> continue scalar path without re-entering SIMD again.
>> > - This gives from 10% to 30% speedups depending on the weight of
>> special characters in the attribute, we don't lose anything here since it
>> advances with SIMD until it can't (using the previous scripts: 1/3, 2/3
>> specials chars).
>> >
>> > 2) Do SIMD path, then use scalar path when we hit a special character,
>> keep re-entering the SIMD path each time.
>> > - This is equivalent to the COPY FROM story, we'll need to find the
>> same heuristic to use for both COPY FROM/TO to reduce the regressions (same
>> regressions: around from 20% to 30% with 1/3, 2/3 specials chars).
>> >
>> > Something else to note is that the scalar path for COPY TO isn't as
>> heavy as the state machine in COPY FROM.
>> >
>> > So if we find the sweet spot for the heuristic, doing the same for COPY
>> TO will be trivial and always beneficial.
>> > Attached is 0004 which is option 1 (SIMD without re-entering), 0005 is
>> the second one.
>>
>> Patches look correct to me. I think we could move these SIMD code
>> portions into a shared function to remove duplication, although that
>> might have a performance impact. I have not benchmarked these patches
>> yet.
>>
>> Another consideration is that these patches might need their own
>> thread, though I am not completely sure about this yet.
>>
>> One question: what do you think about having a 0004-style approach for
>> COPY FROM? What I have in mind is running SIMD for each line & column,
>> stopping SIMD once it can no longer skip an entire chunk, and then
>> continuing with the next line & column.
>>
>> --
>> Regards,
>> Nazir Bilal Yavuz
>> Microsoft
>>
>
> Hello, Nazir, I tried your suggested cpupower commands as well as
> disabling turbo, and my results are indeed more uniform. (see attached
> screenshot of my spreadsheet).
>
> This time, I ran the tests on my Tower PC instead of on my laptop.
>
> I also followed Mark Wong's advice and used the taskset command to pin my
> postgres postmaster (and all of its children) to a single cpu core.
>
> So when I start postgres, I do this to pin it to core 27:
>
> ${PGHOME}/bin/pg_ctl -D ${PGHOME}/data -l ${PGHOME}/logfile.txt start
> PGPID=$(head -1 ${PGHOME}/data/postmaster.pid)
> taskset --cpu-list -p 27 ${PGPID}
>
>
> My results seem similar to yours:
>
> master: Nazir 85ddcc2f4c | Manni 877ae5db
>
> text, no special: 102294 | 302651
> text, 1/3 special: 108946 | 326208
> csv, no special: 121831 | 348930
> csv, 1/3 special: 140063 | 439786
>
> v3
>
> text, no special: 88890 (13.1% speedup) | 227874 (24.7% speedup)
> text, 1/3 special: 110463 (1.4% regression) | 322637 (1.1% speedup)
> csv, no special: 89781 (26.3% speedup) | 226525 (35.1% speedup)
> csv, 1/3 special: 147094 (5.0% regression) | 461501 (4.9% regression)
>
> v4.2
>
> text, no special: 87785 (14.2% speedup) | 225702 (25.4% speedup)
> text, 1/3 special: 127008 (16.6% regression) | 343480 (5.3% regression)
> csv, no special: 88093 (27.7% speedup) | 226633 (35.0% speedup)
> csv, 1/3 special: 164487 (17.4% regression) | 510954 (16.2% regression)
>
> It would seem that both your results and mine show a more serious
> worst-case regression for the v4.2 patches than for the v3 patches. It
> seems also that the speedups for v4.2 and v3 are similar.
>
> I'm currently working with Mark Wong to see if his results continue to be
> dissimilar (as they currently are now) and, if so, why.
> --
> -- Manni Wood EDB: https://www.enterprisedb.com
>
Hello, all.
Now that I am following Nazir's on how to configure my CPU for performance
test run, and now that I am following Mark's advice on pinning the
postmaster to a particular CPU core, I figured I would share the scripts I
have been using to build, run, and test Postges with various patches
applied: https://github.com/manniwood/copysimdperf
With Nazir and Mark's tips, I have seen more consistent numbers on my tower
PC, as shared in a previous e-mail. But Mark and I saw rather variable
results on a different Linux system he has access to. So this has inspired
me to spin up an AWS EC2 instance and test that when I find the time. And
maybe re-test on my Linux laptop.
If anybody else is inspired to test on different setups, that would be
great.
--
-- Manni Wood EDB: https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-12 20:42 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Mark Wong <markwkm@gmail.com>
2025-12-12 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-18 07:35 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-12-24 15:07 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-12-31 13:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-06 20:05 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-01-07 19:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
@ 2026-01-08 20:49 ` Manni Wood <manni.wood@enterprisedb.com>
2026-01-09 00:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-01-09 14:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
1 sibling, 2 replies; 178+ messages in thread
From: Manni Wood @ 2026-01-08 20:49 UTC (permalink / raw)
To: Nazir Bilal Yavuz <byavuz81@gmail.com>; +Cc: KAZAR Ayoub <ma_kazar@esi.dz>; Mark Wong <markwkm@gmail.com>; Nathan Bossart <nathandbossart@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
On Wed, Jan 7, 2026 at 1:13 PM Manni Wood <manni.wood@enterprisedb.com>
wrote:
>
>
> On Tue, Jan 6, 2026 at 2:05 PM Manni Wood <manni.wood@enterprisedb.com>
> wrote:
>
>>
>>
>> On Wed, Dec 31, 2025 at 7:04 AM Nazir Bilal Yavuz <byavuz81@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> On Wed, 24 Dec 2025 at 18:08, KAZAR Ayoub <ma_kazar@esi.dz> wrote:
>>> >
>>> > Hello,
>>> > Following the same path of optimizing COPY FROM using SIMD, i found
>>> that COPY TO can also benefit from this.
>>> >
>>> > I attached a small patch that uses SIMD to skip data and advance as
>>> far as the first special character is found, then fallback to scalar
>>> processing for that character and re-enter the SIMD path again...
>>> > There's two ways to do this:
>>> > 1) Essentially we do SIMD until we find a special character, then
>>> continue scalar path without re-entering SIMD again.
>>> > - This gives from 10% to 30% speedups depending on the weight of
>>> special characters in the attribute, we don't lose anything here since it
>>> advances with SIMD until it can't (using the previous scripts: 1/3, 2/3
>>> specials chars).
>>> >
>>> > 2) Do SIMD path, then use scalar path when we hit a special character,
>>> keep re-entering the SIMD path each time.
>>> > - This is equivalent to the COPY FROM story, we'll need to find the
>>> same heuristic to use for both COPY FROM/TO to reduce the regressions (same
>>> regressions: around from 20% to 30% with 1/3, 2/3 specials chars).
>>> >
>>> > Something else to note is that the scalar path for COPY TO isn't as
>>> heavy as the state machine in COPY FROM.
>>> >
>>> > So if we find the sweet spot for the heuristic, doing the same for
>>> COPY TO will be trivial and always beneficial.
>>> > Attached is 0004 which is option 1 (SIMD without re-entering), 0005 is
>>> the second one.
>>>
>>> Patches look correct to me. I think we could move these SIMD code
>>> portions into a shared function to remove duplication, although that
>>> might have a performance impact. I have not benchmarked these patches
>>> yet.
>>>
>>> Another consideration is that these patches might need their own
>>> thread, though I am not completely sure about this yet.
>>>
>>> One question: what do you think about having a 0004-style approach for
>>> COPY FROM? What I have in mind is running SIMD for each line & column,
>>> stopping SIMD once it can no longer skip an entire chunk, and then
>>> continuing with the next line & column.
>>>
>>> --
>>> Regards,
>>> Nazir Bilal Yavuz
>>> Microsoft
>>>
>>
>> Hello, Nazir, I tried your suggested cpupower commands as well as
>> disabling turbo, and my results are indeed more uniform. (see attached
>> screenshot of my spreadsheet).
>>
>> This time, I ran the tests on my Tower PC instead of on my laptop.
>>
>> I also followed Mark Wong's advice and used the taskset command to pin my
>> postgres postmaster (and all of its children) to a single cpu core.
>>
>> So when I start postgres, I do this to pin it to core 27:
>>
>> ${PGHOME}/bin/pg_ctl -D ${PGHOME}/data -l ${PGHOME}/logfile.txt start
>> PGPID=$(head -1 ${PGHOME}/data/postmaster.pid)
>> taskset --cpu-list -p 27 ${PGPID}
>>
>>
>> My results seem similar to yours:
>>
>> master: Nazir 85ddcc2f4c | Manni 877ae5db
>>
>> text, no special: 102294 | 302651
>> text, 1/3 special: 108946 | 326208
>> csv, no special: 121831 | 348930
>> csv, 1/3 special: 140063 | 439786
>>
>> v3
>>
>> text, no special: 88890 (13.1% speedup) | 227874 (24.7% speedup)
>> text, 1/3 special: 110463 (1.4% regression) | 322637 (1.1% speedup)
>> csv, no special: 89781 (26.3% speedup) | 226525 (35.1% speedup)
>> csv, 1/3 special: 147094 (5.0% regression) | 461501 (4.9% regression)
>>
>> v4.2
>>
>> text, no special: 87785 (14.2% speedup) | 225702 (25.4% speedup)
>> text, 1/3 special: 127008 (16.6% regression) | 343480 (5.3% regression)
>> csv, no special: 88093 (27.7% speedup) | 226633 (35.0% speedup)
>> csv, 1/3 special: 164487 (17.4% regression) | 510954 (16.2% regression)
>>
>> It would seem that both your results and mine show a more serious
>> worst-case regression for the v4.2 patches than for the v3 patches. It
>> seems also that the speedups for v4.2 and v3 are similar.
>>
>> I'm currently working with Mark Wong to see if his results continue to be
>> dissimilar (as they currently are now) and, if so, why.
>> --
>> -- Manni Wood EDB: https://www.enterprisedb.com
>>
>
> Hello, all.
>
> Now that I am following Nazir's on how to configure my CPU for performance
> test run, and now that I am following Mark's advice on pinning the
> postmaster to a particular CPU core, I figured I would share the scripts I
> have been using to build, run, and test Postges with various patches
> applied: https://github.com/manniwood/copysimdperf
>
> With Nazir and Mark's tips, I have seen more consistent numbers on my
> tower PC, as shared in a previous e-mail. But Mark and I saw rather
> variable results on a different Linux system he has access to. So this has
> inspired me to spin up an AWS EC2 instance and test that when I find the
> time. And maybe re-test on my Linux laptop.
>
> If anybody else is inspired to test on different setups, that would be
> great.
> --
> -- Manni Wood EDB: https://www.enterprisedb.com
>
I tested master (bfb335d) and v3 and v4.2 patches on an amazon ec2 instance
(t2.small) and, with Mark's help, proved that on such a small system with
default storage configured, IO will be the bottleneck and the v3 and v4.2
patches show no significant differences over master because the CPU is
always waiting on IO. This is presumably an experience Postgres users will
have when running on systems with IO so slow that the CPU is always waiting
for data.
I went in the other direction and tested an all-RAM setup on my tower PC. I
put the entire data dir in RAM for each postgres instance (master, v3
patch, v4.2 patch), and wrote and copied the test copyfiles from RAM. On
Linux, /ram/user/<myuserid> is tmpfs (ramdisk), so I just put everything
there. I had to shrink the data sizes compared to previous runs (to not run
out of ramdisk space) but Nazir's cpupower tips are making all of my test
runs much more uniform, so I no longer feel that I need huge data sizes to
get good results.
Here are the results when all of the files are on RAM disks:
master: bfb335df
text, no special: 30372
text, 1/3 special: 32665
csv, no special: 34925
csv, 1/3 special: 44044
v3
text, no special: 22840 (24.7% speedup)
text, 1/3 special: 32448 (0.6% speedup)
csv, no special: 22642 (35.1% speedup)
csv, 1/3 special: 46280 (5.1% regression)
v4.2
text, no special: 22677 (25.3% speedup)
text, 1/3 special: 34512 (6.5% regression)
csv, no special: 22686 (35.0% speedup)
csv, 1/3 special: 51411 (16.7% regression)
Assuming all-storage-is-RAM setups get us closer to the theoretical limit
of each patch, it looks like v3 holds up quite well to v4.2 in the best
case scenarios, while v3 has better performance than v4.2 in the worst-case
scenarios.
Let me know what you think!
--
-- Manni Wood EDB: https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-12 20:42 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Mark Wong <markwkm@gmail.com>
2025-12-12 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-18 07:35 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-12-24 15:07 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-12-31 13:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-06 20:05 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-01-07 19:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-01-08 20:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
@ 2026-01-09 00:01 ` Manni Wood <manni.wood@enterprisedb.com>
1 sibling, 0 replies; 178+ messages in thread
From: Manni Wood @ 2026-01-09 00:01 UTC (permalink / raw)
To: Nazir Bilal Yavuz <byavuz81@gmail.com>; +Cc: KAZAR Ayoub <ma_kazar@esi.dz>; Mark Wong <markwkm@gmail.com>; Nathan Bossart <nathandbossart@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
On Thu, Jan 8, 2026 at 2:49 PM Manni Wood <manni.wood@enterprisedb.com>
wrote:
>
>
> On Wed, Jan 7, 2026 at 1:13 PM Manni Wood <manni.wood@enterprisedb.com>
> wrote:
>
>>
>>
>> On Tue, Jan 6, 2026 at 2:05 PM Manni Wood <manni.wood@enterprisedb.com>
>> wrote:
>>
>>>
>>>
>>> On Wed, Dec 31, 2025 at 7:04 AM Nazir Bilal Yavuz <byavuz81@gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> On Wed, 24 Dec 2025 at 18:08, KAZAR Ayoub <ma_kazar@esi.dz> wrote:
>>>> >
>>>> > Hello,
>>>> > Following the same path of optimizing COPY FROM using SIMD, i found
>>>> that COPY TO can also benefit from this.
>>>> >
>>>> > I attached a small patch that uses SIMD to skip data and advance as
>>>> far as the first special character is found, then fallback to scalar
>>>> processing for that character and re-enter the SIMD path again...
>>>> > There's two ways to do this:
>>>> > 1) Essentially we do SIMD until we find a special character, then
>>>> continue scalar path without re-entering SIMD again.
>>>> > - This gives from 10% to 30% speedups depending on the weight of
>>>> special characters in the attribute, we don't lose anything here since it
>>>> advances with SIMD until it can't (using the previous scripts: 1/3, 2/3
>>>> specials chars).
>>>> >
>>>> > 2) Do SIMD path, then use scalar path when we hit a special
>>>> character, keep re-entering the SIMD path each time.
>>>> > - This is equivalent to the COPY FROM story, we'll need to find the
>>>> same heuristic to use for both COPY FROM/TO to reduce the regressions (same
>>>> regressions: around from 20% to 30% with 1/3, 2/3 specials chars).
>>>> >
>>>> > Something else to note is that the scalar path for COPY TO isn't as
>>>> heavy as the state machine in COPY FROM.
>>>> >
>>>> > So if we find the sweet spot for the heuristic, doing the same for
>>>> COPY TO will be trivial and always beneficial.
>>>> > Attached is 0004 which is option 1 (SIMD without re-entering), 0005
>>>> is the second one.
>>>>
>>>> Patches look correct to me. I think we could move these SIMD code
>>>> portions into a shared function to remove duplication, although that
>>>> might have a performance impact. I have not benchmarked these patches
>>>> yet.
>>>>
>>>> Another consideration is that these patches might need their own
>>>> thread, though I am not completely sure about this yet.
>>>>
>>>> One question: what do you think about having a 0004-style approach for
>>>> COPY FROM? What I have in mind is running SIMD for each line & column,
>>>> stopping SIMD once it can no longer skip an entire chunk, and then
>>>> continuing with the next line & column.
>>>>
>>>> --
>>>> Regards,
>>>> Nazir Bilal Yavuz
>>>> Microsoft
>>>>
>>>
>>> Hello, Nazir, I tried your suggested cpupower commands as well as
>>> disabling turbo, and my results are indeed more uniform. (see attached
>>> screenshot of my spreadsheet).
>>>
>>> This time, I ran the tests on my Tower PC instead of on my laptop.
>>>
>>> I also followed Mark Wong's advice and used the taskset command to pin
>>> my postgres postmaster (and all of its children) to a single cpu core.
>>>
>>> So when I start postgres, I do this to pin it to core 27:
>>>
>>> ${PGHOME}/bin/pg_ctl -D ${PGHOME}/data -l ${PGHOME}/logfile.txt start
>>> PGPID=$(head -1 ${PGHOME}/data/postmaster.pid)
>>> taskset --cpu-list -p 27 ${PGPID}
>>>
>>>
>>> My results seem similar to yours:
>>>
>>> master: Nazir 85ddcc2f4c | Manni 877ae5db
>>>
>>> text, no special: 102294 | 302651
>>> text, 1/3 special: 108946 | 326208
>>> csv, no special: 121831 | 348930
>>> csv, 1/3 special: 140063 | 439786
>>>
>>> v3
>>>
>>> text, no special: 88890 (13.1% speedup) | 227874 (24.7% speedup)
>>> text, 1/3 special: 110463 (1.4% regression) | 322637 (1.1% speedup)
>>> csv, no special: 89781 (26.3% speedup) | 226525 (35.1% speedup)
>>> csv, 1/3 special: 147094 (5.0% regression) | 461501 (4.9% regression)
>>>
>>> v4.2
>>>
>>> text, no special: 87785 (14.2% speedup) | 225702 (25.4% speedup)
>>> text, 1/3 special: 127008 (16.6% regression) | 343480 (5.3% regression)
>>> csv, no special: 88093 (27.7% speedup) | 226633 (35.0% speedup)
>>> csv, 1/3 special: 164487 (17.4% regression) | 510954 (16.2% regression)
>>>
>>> It would seem that both your results and mine show a more serious
>>> worst-case regression for the v4.2 patches than for the v3 patches. It
>>> seems also that the speedups for v4.2 and v3 are similar.
>>>
>>> I'm currently working with Mark Wong to see if his results continue to
>>> be dissimilar (as they currently are now) and, if so, why.
>>> --
>>> -- Manni Wood EDB: https://www.enterprisedb.com
>>>
>>
>> Hello, all.
>>
>> Now that I am following Nazir's on how to configure my CPU for
>> performance test run, and now that I am following Mark's advice on pinning
>> the postmaster to a particular CPU core, I figured I would share the
>> scripts I have been using to build, run, and test Postges with various
>> patches applied: https://github.com/manniwood/copysimdperf
>>
>> With Nazir and Mark's tips, I have seen more consistent numbers on my
>> tower PC, as shared in a previous e-mail. But Mark and I saw rather
>> variable results on a different Linux system he has access to. So this has
>> inspired me to spin up an AWS EC2 instance and test that when I find the
>> time. And maybe re-test on my Linux laptop.
>>
>> If anybody else is inspired to test on different setups, that would be
>> great.
>> --
>> -- Manni Wood EDB: https://www.enterprisedb.com
>>
>
> I tested master (bfb335d) and v3 and v4.2 patches on an amazon ec2
> instance (t2.small) and, with Mark's help, proved that on such a small
> system with default storage configured, IO will be the bottleneck and the
> v3 and v4.2 patches show no significant differences over master because the
> CPU is always waiting on IO. This is presumably an experience Postgres
> users will have when running on systems with IO so slow that the CPU is
> always waiting for data.
>
> I went in the other direction and tested an all-RAM setup on my tower PC.
> I put the entire data dir in RAM for each postgres instance (master, v3
> patch, v4.2 patch), and wrote and copied the test copyfiles from RAM. On
> Linux, /ram/user/<myuserid> is tmpfs (ramdisk), so I just put everything
> there. I had to shrink the data sizes compared to previous runs (to not run
> out of ramdisk space) but Nazir's cpupower tips are making all of my test
> runs much more uniform, so I no longer feel that I need huge data sizes to
> get good results.
>
> Here are the results when all of the files are on RAM disks:
>
> master: bfb335df
>
> text, no special: 30372
> text, 1/3 special: 32665
> csv, no special: 34925
> csv, 1/3 special: 44044
>
> v3
>
> text, no special: 22840 (24.7% speedup)
> text, 1/3 special: 32448 (0.6% speedup)
> csv, no special: 22642 (35.1% speedup)
> csv, 1/3 special: 46280 (5.1% regression)
>
> v4.2
>
> text, no special: 22677 (25.3% speedup)
> text, 1/3 special: 34512 (6.5% regression)
> csv, no special: 22686 (35.0% speedup)
> csv, 1/3 special: 51411 (16.7% regression)
>
> Assuming all-storage-is-RAM setups get us closer to the theoretical limit
> of each patch, it looks like v3 holds up quite well to v4.2 in the best
> case scenarios, while v3 has better performance than v4.2 in the worst-case
> scenarios.
>
> Let me know what you think!
> --
> -- Manni Wood EDB: https://www.enterprisedb.com
>
Ayoub Kazar, I tested your v4 "copy to" patch, doing everything in RAM, and
using the cpupower tips from above. (I wanted to test your v5, but `git
apply --check` gave me an error, so I can look at that another day.)
The results look great:
master: (forgot to get commit hash)
text, no special: 8165
text, 1/3 special: 22662
csv, no special: 9619
csv, 1/3 special: 23213
v4 (copy to)
text, no special: 4577 (43.9% speedup)
text, 1/3 special: 22847 (0.8% regression)
csv, no special: 4720 (50.9% speedup)
csv, 1/3 special: 23195 (0.07% regression)
Seems like a very clear win to me!
--
-- Manni Wood EDB: https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-12 20:42 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Mark Wong <markwkm@gmail.com>
2025-12-12 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-18 07:35 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-12-24 15:07 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-12-31 13:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-06 20:05 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-01-07 19:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-01-08 20:49 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
@ 2026-01-09 14:22 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
1 sibling, 0 replies; 178+ messages in thread
From: Nazir Bilal Yavuz @ 2026-01-09 14:22 UTC (permalink / raw)
To: Manni Wood <manni.wood@enterprisedb.com>; +Cc: KAZAR Ayoub <ma_kazar@esi.dz>; Mark Wong <markwkm@gmail.com>; Nathan Bossart <nathandbossart@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
Hi,
On Thu, 8 Jan 2026 at 23:50, Manni Wood <manni.wood@enterprisedb.com> wrote:
>
> I tested master (bfb335d) and v3 and v4.2 patches on an amazon ec2 instance (t2.small) and, with Mark's help, proved that on such a small system with default storage configured, IO will be the bottleneck and the v3 and v4.2 patches show no significant differences over master because the CPU is always waiting on IO. This is presumably an experience Postgres users will have when running on systems with IO so slow that the CPU is always waiting for data.
I think this is expected and acceptable since there is no regression.
> I went in the other direction and tested an all-RAM setup on my tower PC. I put the entire data dir in RAM for each postgres instance (master, v3 patch, v4.2 patch), and wrote and copied the test copyfiles from RAM. On Linux, /ram/user/<myuserid> is tmpfs (ramdisk), so I just put everything there. I had to shrink the data sizes compared to previous runs (to not run out of ramdisk space) but Nazir's cpupower tips are making all of my test runs much more uniform, so I no longer feel that I need huge data sizes to get good results.
This is an informative scenario to test, thank you for doing this.
> Here are the results when all of the files are on RAM disks:
>
> master: bfb335df
>
> text, no special: 30372
> text, 1/3 special: 32665
> csv, no special: 34925
> csv, 1/3 special: 44044
>
> v3
>
> text, no special: 22840 (24.7% speedup)
> text, 1/3 special: 32448 (0.6% speedup)
> csv, no special: 22642 (35.1% speedup)
> csv, 1/3 special: 46280 (5.1% regression)
>
> v4.2
>
> text, no special: 22677 (25.3% speedup)
> text, 1/3 special: 34512 (6.5% regression)
> csv, no special: 22686 (35.0% speedup)
> csv, 1/3 special: 51411 (16.7% regression)
>
> Assuming all-storage-is-RAM setups get us closer to the theoretical limit of each patch, it looks like v3 holds up quite well to v4.2 in the best case scenarios, while v3 has better performance than v4.2 in the worst-case scenarios.!
I agree with you, I think all-storage-is-RAM is the best scenario to
benchmark regressions.
Your all-storage-is-RAM benchmark results are similar to your normal
benchmark here [1]. I expect the regressions in the all-storage-is-RAM
results to be worse than [1] since the effect of SIMD will be more
visible as there is no IO wait. What do you think about that?
[1] https://postgr.es/m/CAKWEB6qQmPejLwndzMcVqhO%2Bu9vgUf44kYAKg4QtyG-%3DKH%3DYGg%40mail.gmail.com
--
Regards,
Nazir Bilal Yavuz
Microsoft
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-12 20:42 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Mark Wong <markwkm@gmail.com>
2025-12-12 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-18 07:35 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-12-24 15:07 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-12-31 13:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-06 20:05 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-01-07 19:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
@ 2026-01-09 14:21 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-13 19:12 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Mark Wong <markwkm@gmail.com>
1 sibling, 1 reply; 178+ messages in thread
From: Nazir Bilal Yavuz @ 2026-01-09 14:21 UTC (permalink / raw)
To: Manni Wood <manni.wood@enterprisedb.com>; +Cc: KAZAR Ayoub <ma_kazar@esi.dz>; Mark Wong <markwkm@gmail.com>; Nathan Bossart <nathandbossart@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
Hi,
On Wed, 7 Jan 2026 at 22:13, Manni Wood <manni.wood@enterprisedb.com> wrote:
>
> Now that I am following Nazir's on how to configure my CPU for performance test run, and now that I am following Mark's advice on pinning the postmaster to a particular CPU core, I figured I would share the scripts I have been using to build, run, and test Postges with various patches applied: https://github.com/manniwood/copysimdperf
Thank you for sharing this! I will try to use it soon.
> With Nazir and Mark's tips, I have seen more consistent numbers on my tower PC, as shared in a previous e-mail. But Mark and I saw rather variable results on a different Linux system he has access to. So this has inspired me to spin up an AWS EC2 instance and test that when I find the time. And maybe re-test on my Linux laptop.
Were you able to understand why Mark's benchmark results are different
from ours?
--
Regards,
Nazir Bilal Yavuz
Microsoft
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-12 20:42 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Mark Wong <markwkm@gmail.com>
2025-12-12 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-18 07:35 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-12-24 15:07 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-12-31 13:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-06 20:05 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-01-07 19:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-01-09 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
@ 2026-01-13 19:12 ` Mark Wong <markwkm@gmail.com>
2026-01-14 00:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
0 siblings, 1 reply; 178+ messages in thread
From: Mark Wong @ 2026-01-13 19:12 UTC (permalink / raw)
To: Nazir Bilal Yavuz <byavuz81@gmail.com>; +Cc: Manni Wood <manni.wood@enterprisedb.com>; KAZAR Ayoub <ma_kazar@esi.dz>; Nathan Bossart <nathandbossart@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
On Fri, Jan 09, 2026 at 05:21:45PM +0300, Nazir Bilal Yavuz wrote:
> Were you able to understand why Mark's benchmark results are different
> from ours?
Not yet... I had some guesses, which is why I suggested the processor pinning
and using a ramdisk. But we haven't tried applying all of those to my laptop,
which has 3 core types, or the POWER system, which may be interesting to use a
ram disk on.
I'm curious though, and admittedly haven't tried looking myself yet, about how
the SIMD calls might look across different processor architectures. We'll try
to get that on the POWER system soon...
Regards,
Mark
--
Mark Wong <markwkm@gmail.com>
EDB https://enterprisedb.com
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-12 20:42 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Mark Wong <markwkm@gmail.com>
2025-12-12 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-18 07:35 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-12-24 15:07 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-12-31 13:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-06 20:05 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-01-07 19:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-01-09 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-13 19:12 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Mark Wong <markwkm@gmail.com>
@ 2026-01-14 00:20 ` Manni Wood <manni.wood@enterprisedb.com>
2026-01-17 21:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-01-30 22:05 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Mark Wong <markwkm@gmail.com>
0 siblings, 2 replies; 178+ messages in thread
From: Manni Wood @ 2026-01-14 00:20 UTC (permalink / raw)
To: Mark Wong <markwkm@gmail.com>; +Cc: Nazir Bilal Yavuz <byavuz81@gmail.com>; KAZAR Ayoub <ma_kazar@esi.dz>; Nathan Bossart <nathandbossart@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
On Tue, Jan 13, 2026 at 1:12 PM Mark Wong <markwkm@gmail.com> wrote:
> On Fri, Jan 09, 2026 at 05:21:45PM +0300, Nazir Bilal Yavuz wrote:
> > Were you able to understand why Mark's benchmark results are different
> > from ours?
>
> Not yet... I had some guesses, which is why I suggested the processor
> pinning
> and using a ramdisk. But we haven't tried applying all of those to my
> laptop,
> which has 3 core types, or the POWER system, which may be interesting to
> use a
> ram disk on.
>
> I'm curious though, and admittedly haven't tried looking myself yet, about
> how
> the SIMD calls might look across different processor architectures. We'll
> try
> to get that on the POWER system soon...
>
> Regards,
> Mark
> --
> Mark Wong <markwkm@gmail.com>
> EDB https://enterprisedb.com
>
Hello!
Nazir, I'm glad you are finding the benchmarks useful. I have more! :-)
All of these benchmarks are all-in-RAM, because I do think that is the best
way of getting closest to the theoretical best and worst case scenarios.
My laptop:
master: (852558b9)
text, no special: 14996
text, 1/3 special: 17270
csv, no special: 18274
csv, 1/3 special: 23852
v3
text, no special: 11282 (24.7% speedup)
text, 1/3 special: 15748 (8.8% speedup) <-- I don't believe this but it's
what I got
csv, no special: 11571 (36.6% speedup)
csv, 1/3 special: 19934 (16.4% speedup) <-- I don't believe this but it's
what I got
v4.2
text, no special: 11139 (25.7% speedup)
text, 1/3 special: 18900 (9.4% regression)
csv, no special: 11490 (37.1% speedup)
csv, 1/3 special: 26134 (9.5% regression)
An AWS EC2 t2.2xlarge instance
master: (852558b9)
text, no special: 20677
text, 1/3 special: 22660
csv, no special: 24534
csv, 1/3 special: 30999
v3
text, no special: 17534 (15.2% speedup)
text, 1/3 special: 22816 (0.6% regression)
csv, no special: 17664 (28.0% speedup)
csv, 1/3 special: 29338 (5.3% speedup) <-- I don't believe this but it's
what I got
v4.2
text, no special: 17459 (15.5% speedup)
text, 1/3 special: 25051 (10.5% regression)
csv, no special: 17574 (28.3% speedup)
csv, 1/3 special: 32092 (3.5% regression)
An AWS EC2 t4g.2xlarge instance (using ARM processor; first test of ARM
processor!)
master: (852558b9)
text, no special: 22081
text, 1/3 special: 25100
csv, no special: 27296
csv, 1/3 special: 32344
v3
text, no special: 17724 (19.7% speedup)
text, 1/3 special: 27606 (9.9% regression) <-- yikes! We would want to test
this more
csv, no special: 17597 (35.5% speedup)
csv, 1/3 special: 32597 (0.8% regression)
v4.2
text, no special: 17674 (20% speedup)
text, 1/3 special: 25773 (2.6% regression) <-- this regression is less than
for the v3 patch? Atypical...
csv, no special: 17651 (35.3% speedup)
csv, 1/3 special: 34055 (5.3% regression)
Yes, I think I agree with you that the everything-in-RAM benchmarks will
make the regressions more pronounced, just like the everything-in-RAM
benchmarks make the improvements more pronounced.
I am not sure why the CSV regression, compared to the TXT regression (even
for the v3 patch which has smaller regressions than the v4.2 patch) is
usually worse. I probably should look over some flame graphs and see if I
can find the place where the CSV-parsing code is so much slower. The CSV
regression is actually a bit frustrating (at around 5%) because the TXT
regression, at less than 1% (for the v3 patch) is so much easier to bare.
Here are some copy-to benchmarks for the v4 patch that applies SIMD to the
copy-to code.
These were all-in-RAM tests.
My laptop
master: (852558b9)
text, no special: 2948
text, 1/3 special: 11258
csv, no special: 6245
csv, 1/3 special: 11258
v4 (copy to)
text, no special: 2126 (27.9% speedup)
text, 1/3 special: 12080 (7.3% regression) <-- did not see such a big
regression before
csv, no special: 2432 (61.0% speedup)
csv, 1/3 special: 12344 (4.0% regression) <-- did not see such a big
regression before
An AWS EC2 t2.2xlarge instance
master: (852558b9)
text, no special: 4647
text, 1/3 special: 13865
csv, no special: 5421
csv, 1/3 special: 15284
v4 (copy to)
text, no special: 2460 (47.0% speedup)
text, 1/3 special: 14023 (1.1% regression)
csv, no special: 2667 (50.7% speedup)
csv, 1/3 special: 15251 (0.2% speedup)
An AWS EC2 t4g.2xlarge instance (using ARM processor; first test of ARM
processor!)
master: (852558b9)
text, no special: 6951
text, 1/3 special: 17857
csv, no special: 7951
csv, 1/3 special: 18504
v4 (copy to)
text, no special: 3372 (51.4% speedup)
text, 1/3 special: 15713 (12.0% speedup)
csv, no special: 3233 (59.3% speedup)
csv, 1/3 special: 1622 (12.3% speedup)
Once again, the v4 patch for copy-to seems like a clearer win, though, to
be fair, there were regressions when running on my laptop. (I'm starting to
think servers or desktops are better than laptops for testing these things,
though maybe that's my bias: it just seems like the server results are
always less surprising.)
Hope you all continue to find these useful...
--
-- Manni Wood EDB: https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-12 20:42 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Mark Wong <markwkm@gmail.com>
2025-12-12 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-18 07:35 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-12-24 15:07 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-12-31 13:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-06 20:05 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-01-07 19:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-01-09 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-13 19:12 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Mark Wong <markwkm@gmail.com>
2026-01-14 00:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
@ 2026-01-17 21:24 ` KAZAR Ayoub <ma_kazar@esi.dz>
2026-01-20 20:48 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
1 sibling, 1 reply; 178+ messages in thread
From: KAZAR Ayoub @ 2026-01-17 21:24 UTC (permalink / raw)
To: Manni Wood <manni.wood@enterprisedb.com>; +Cc: Mark Wong <markwkm@gmail.com>; Nazir Bilal Yavuz <byavuz81@gmail.com>; Nathan Bossart <nathandbossart@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
Hello,
Thank you for these benchmarks, this is helpful !
On Wed, Jan 14, 2026 at 1:20 AM Manni Wood <manni.wood@enterprisedb.com>
wrote:
>
>
> Hello!
>
> Nazir, I'm glad you are finding the benchmarks useful. I have more! :-)
>
> All of these benchmarks are all-in-RAM, because I do think that is the
> best way of getting closest to the theoretical best and worst case
> scenarios.
>
> My laptop:
>
> master: (852558b9)
>
> text, no special: 14996
> text, 1/3 special: 17270
> csv, no special: 18274
> csv, 1/3 special: 23852
>
> v3
>
> text, no special: 11282 (24.7% speedup)
> text, 1/3 special: 15748 (8.8% speedup) <-- I don't believe this but it's
> what I got
> csv, no special: 11571 (36.6% speedup)
> csv, 1/3 special: 19934 (16.4% speedup) <-- I don't believe this but it's
> what I got
>
> v4.2
>
> text, no special: 11139 (25.7% speedup)
> text, 1/3 special: 18900 (9.4% regression)
> csv, no special: 11490 (37.1% speedup)
> csv, 1/3 special: 26134 (9.5% regression)
>
> An AWS EC2 t2.2xlarge instance
>
> master: (852558b9)
>
> text, no special: 20677
> text, 1/3 special: 22660
> csv, no special: 24534
> csv, 1/3 special: 30999
>
> v3
>
> text, no special: 17534 (15.2% speedup)
> text, 1/3 special: 22816 (0.6% regression)
> csv, no special: 17664 (28.0% speedup)
> csv, 1/3 special: 29338 (5.3% speedup) <-- I don't believe this but it's
> what I got
>
> v4.2
>
> text, no special: 17459 (15.5% speedup)
> text, 1/3 special: 25051 (10.5% regression)
> csv, no special: 17574 (28.3% speedup)
> csv, 1/3 special: 32092 (3.5% regression)
>
> An AWS EC2 t4g.2xlarge instance (using ARM processor; first test of ARM
> processor!)
>
> master: (852558b9)
>
> text, no special: 22081
> text, 1/3 special: 25100
> csv, no special: 27296
> csv, 1/3 special: 32344
>
> v3
>
> text, no special: 17724 (19.7% speedup)
> text, 1/3 special: 27606 (9.9% regression) <-- yikes! We would want to
> test this more
> csv, no special: 17597 (35.5% speedup)
> csv, 1/3 special: 32597 (0.8% regression)
>
> v4.2
>
> text, no special: 17674 (20% speedup)
> text, 1/3 special: 25773 (2.6% regression) <-- this regression is less
> than for the v3 patch? Atypical...
> csv, no special: 17651 (35.3% speedup)
> csv, 1/3 special: 34055 (5.3% regression)
>
> Yes, I think I agree with you that the everything-in-RAM benchmarks will
> make the regressions more pronounced, just like the everything-in-RAM
> benchmarks make the improvements more pronounced.
>
> I am not sure why the CSV regression, compared to the TXT regression (even
> for the v3 patch which has smaller regressions than the v4.2 patch) is
> usually worse. I probably should look over some flame graphs and see if I
> can find the place where the CSV-parsing code is so much slower. The CSV
> regression is actually a bit frustrating (at around 5%) because the TXT
> regression, at less than 1% (for the v3 patch) is so much easier to bare.
>
The only reasons that i can think of for this problem is the CSV state
machine is more complex than TEXT, which might imply that for everything
related to branch prediction, stalls ..etc becomes more demanding in CSV
mode, i see this by previous tight micro benchmarks on CopyReadLineText, it
has tiny less IPC, more branch misses, stalls and i assume instruction
cache misses shouldn't be a problem since the generated code duplicates the
scalar path, also the code for it isn't that large for one core instruction
cache anyways (mine has 5.8KB per core).
We can use perf_event_open[1] around CopyReadLine to see what's going on
exactly with the counters if someone wants to confirm.
>
>
> Here are some copy-to benchmarks for the v4 patch that applies SIMD to the
> copy-to code.
>
> These were all-in-RAM tests.
>
> My laptop
>
> master: (852558b9)
>
> text, no special: 2948
> text, 1/3 special: 11258
> csv, no special: 6245
> csv, 1/3 special: 11258
>
> v4 (copy to)
>
> text, no special: 2126 (27.9% speedup)
> text, 1/3 special: 12080 (7.3% regression) <-- did not see such a big
> regression before
> csv, no special: 2432 (61.0% speedup)
> csv, 1/3 special: 12344 (4.0% regression) <-- did not see such a big
> regression before
>
> An AWS EC2 t2.2xlarge instance
>
> master: (852558b9)
>
> text, no special: 4647
> text, 1/3 special: 13865
> csv, no special: 5421
> csv, 1/3 special: 15284
>
> v4 (copy to)
>
> text, no special: 2460 (47.0% speedup)
> text, 1/3 special: 14023 (1.1% regression)
> csv, no special: 2667 (50.7% speedup)
> csv, 1/3 special: 15251 (0.2% speedup)
>
> An AWS EC2 t4g.2xlarge instance (using ARM processor; first test of ARM
> processor!)
>
> master: (852558b9)
>
> text, no special: 6951
> text, 1/3 special: 17857
> csv, no special: 7951
> csv, 1/3 special: 18504
>
> v4 (copy to)
>
> text, no special: 3372 (51.4% speedup)
> text, 1/3 special: 15713 (12.0% speedup)
> csv, no special: 3233 (59.3% speedup)
> csv, 1/3 special: 1622 (12.3% speedup)
>
> Once again, the v4 patch for copy-to seems like a clearer win, though, to
> be fair, there were regressions when running on my laptop. (I'm starting to
> think servers or desktops are better than laptops for testing these things,
> though maybe that's my bias: it just seems like the server results are
> always less surprising.)
>
Indeed, I agree on this too.
[1]https://man7.org/linux/man-pages/man2/perf_event_open.2.html
Regards,
Ayoub Kazar
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-12 20:42 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Mark Wong <markwkm@gmail.com>
2025-12-12 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-18 07:35 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-12-24 15:07 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-12-31 13:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-06 20:05 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-01-07 19:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-01-09 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-13 19:12 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Mark Wong <markwkm@gmail.com>
2026-01-14 00:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-01-17 21:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
@ 2026-01-20 20:48 ` Manni Wood <manni.wood@enterprisedb.com>
2026-01-22 18:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
0 siblings, 1 reply; 178+ messages in thread
From: Manni Wood @ 2026-01-20 20:48 UTC (permalink / raw)
To: KAZAR Ayoub <ma_kazar@esi.dz>; +Cc: Mark Wong <markwkm@gmail.com>; Nazir Bilal Yavuz <byavuz81@gmail.com>; Nathan Bossart <nathandbossart@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
On Sat, Jan 17, 2026 at 3:25 PM KAZAR Ayoub <ma_kazar@esi.dz> wrote:
> Hello,
> Thank you for these benchmarks, this is helpful !
>
> On Wed, Jan 14, 2026 at 1:20 AM Manni Wood <manni.wood@enterprisedb.com>
> wrote:
>
>>
>>
>> Hello!
>>
>> Nazir, I'm glad you are finding the benchmarks useful. I have more! :-)
>>
>> All of these benchmarks are all-in-RAM, because I do think that is the
>> best way of getting closest to the theoretical best and worst case
>> scenarios.
>>
>> My laptop:
>>
>> master: (852558b9)
>>
>> text, no special: 14996
>> text, 1/3 special: 17270
>> csv, no special: 18274
>> csv, 1/3 special: 23852
>>
>> v3
>>
>> text, no special: 11282 (24.7% speedup)
>> text, 1/3 special: 15748 (8.8% speedup) <-- I don't believe this but it's
>> what I got
>> csv, no special: 11571 (36.6% speedup)
>> csv, 1/3 special: 19934 (16.4% speedup) <-- I don't believe this but it's
>> what I got
>>
>> v4.2
>>
>> text, no special: 11139 (25.7% speedup)
>> text, 1/3 special: 18900 (9.4% regression)
>> csv, no special: 11490 (37.1% speedup)
>> csv, 1/3 special: 26134 (9.5% regression)
>>
>> An AWS EC2 t2.2xlarge instance
>>
>> master: (852558b9)
>>
>> text, no special: 20677
>> text, 1/3 special: 22660
>> csv, no special: 24534
>> csv, 1/3 special: 30999
>>
>> v3
>>
>> text, no special: 17534 (15.2% speedup)
>> text, 1/3 special: 22816 (0.6% regression)
>> csv, no special: 17664 (28.0% speedup)
>> csv, 1/3 special: 29338 (5.3% speedup) <-- I don't believe this but it's
>> what I got
>>
>> v4.2
>>
>> text, no special: 17459 (15.5% speedup)
>> text, 1/3 special: 25051 (10.5% regression)
>> csv, no special: 17574 (28.3% speedup)
>> csv, 1/3 special: 32092 (3.5% regression)
>>
>> An AWS EC2 t4g.2xlarge instance (using ARM processor; first test of ARM
>> processor!)
>>
>> master: (852558b9)
>>
>> text, no special: 22081
>> text, 1/3 special: 25100
>> csv, no special: 27296
>> csv, 1/3 special: 32344
>>
>> v3
>>
>> text, no special: 17724 (19.7% speedup)
>> text, 1/3 special: 27606 (9.9% regression) <-- yikes! We would want to
>> test this more
>> csv, no special: 17597 (35.5% speedup)
>> csv, 1/3 special: 32597 (0.8% regression)
>>
>> v4.2
>>
>> text, no special: 17674 (20% speedup)
>> text, 1/3 special: 25773 (2.6% regression) <-- this regression is less
>> than for the v3 patch? Atypical...
>> csv, no special: 17651 (35.3% speedup)
>> csv, 1/3 special: 34055 (5.3% regression)
>>
>> Yes, I think I agree with you that the everything-in-RAM benchmarks will
>> make the regressions more pronounced, just like the everything-in-RAM
>> benchmarks make the improvements more pronounced.
>>
>> I am not sure why the CSV regression, compared to the TXT regression
>> (even for the v3 patch which has smaller regressions than the v4.2 patch)
>> is usually worse. I probably should look over some flame graphs and see if
>> I can find the place where the CSV-parsing code is so much slower. The CSV
>> regression is actually a bit frustrating (at around 5%) because the TXT
>> regression, at less than 1% (for the v3 patch) is so much easier to bare.
>>
> The only reasons that i can think of for this problem is the CSV state
> machine is more complex than TEXT, which might imply that for everything
> related to branch prediction, stalls ..etc becomes more demanding in CSV
> mode, i see this by previous tight micro benchmarks on CopyReadLineText, it
> has tiny less IPC, more branch misses, stalls and i assume instruction
> cache misses shouldn't be a problem since the generated code duplicates the
> scalar path, also the code for it isn't that large for one core instruction
> cache anyways (mine has 5.8KB per core).
> We can use perf_event_open[1] around CopyReadLine to see what's going on
> exactly with the counters if someone wants to confirm.
>
>>
>>
>
>> Here are some copy-to benchmarks for the v4 patch that applies SIMD to
>> the copy-to code.
>>
>> These were all-in-RAM tests.
>>
>> My laptop
>>
>> master: (852558b9)
>>
>> text, no special: 2948
>> text, 1/3 special: 11258
>> csv, no special: 6245
>> csv, 1/3 special: 11258
>>
>> v4 (copy to)
>>
>> text, no special: 2126 (27.9% speedup)
>> text, 1/3 special: 12080 (7.3% regression) <-- did not see such a big
>> regression before
>> csv, no special: 2432 (61.0% speedup)
>> csv, 1/3 special: 12344 (4.0% regression) <-- did not see such a big
>> regression before
>>
>> An AWS EC2 t2.2xlarge instance
>>
>> master: (852558b9)
>>
>> text, no special: 4647
>> text, 1/3 special: 13865
>> csv, no special: 5421
>> csv, 1/3 special: 15284
>>
>> v4 (copy to)
>>
>> text, no special: 2460 (47.0% speedup)
>> text, 1/3 special: 14023 (1.1% regression)
>> csv, no special: 2667 (50.7% speedup)
>> csv, 1/3 special: 15251 (0.2% speedup)
>>
>> An AWS EC2 t4g.2xlarge instance (using ARM processor; first test of ARM
>> processor!)
>>
>> master: (852558b9)
>>
>> text, no special: 6951
>> text, 1/3 special: 17857
>> csv, no special: 7951
>> csv, 1/3 special: 18504
>>
>> v4 (copy to)
>>
>> text, no special: 3372 (51.4% speedup)
>> text, 1/3 special: 15713 (12.0% speedup)
>> csv, no special: 3233 (59.3% speedup)
>> csv, 1/3 special: 1622 (12.3% speedup)
>>
>> Once again, the v4 patch for copy-to seems like a clearer win, though, to
>> be fair, there were regressions when running on my laptop. (I'm starting to
>> think servers or desktops are better than laptops for testing these things,
>> though maybe that's my bias: it just seems like the server results are
>> always less surprising.)
>>
> Indeed, I agree on this too.
>
> [1]https://man7.org/linux/man-pages/man2/perf_event_open.2.html
>
> Regards,
> Ayoub Kazar
>
Hello, all I have more benchmarks.
These benchmarks are from a Raspberry Pi 5 that I bought. It has an Arm
Cortex A76 processor.
(I was so impressed with the stability of the results I got on my
standalone Intel tower PC that I figured I needed a standalone Arm-based
machine that was not a laptop and not a VM at a cloud service provider. The
run-to-run results were indeed more stable, just like with my standalone
tower PC.)
COPY FROM
master: (852558b9)
text, no special: 9111
text, 1/3 special: 10302
csv, no special: 11147
csv, 1/3 special: 13375
v3
text, no special: 7351 (19.3% speedup)
text, 1/3 special: 10397 (0.9% regression)
csv, no special: 7272 (34.7% speedup)
csv, 1/3 special: 13472 (0.7% regression)
v4.2
text, no special: 7300 (19.6% speedup)
text, 1/3 special: 10537 (2.3% regression)
csv, no special: 7260 (34.8% speedup)
csv, 1/3 special: 13881 (3.8% regression)
COPY TO
master: (852558b9)
text, no special: 2446
text, 1/3 special: 6988
csv, no special: 2822
csv, 1/3 special: 6967
v4 (copy to)
text, no special: 1533 (37.3% speedup)
text, 1/3 special: 5949 (14.8% speedup)
csv, no special: 1560 (44.7% speedup)
csv, 1/3 special: 6006 (13.8% speedup)
I find these results particularly exciting because with the COPY FROM v3
patch, the worst-case scenarios are just under 1% regression. The v4 COPY
TO patch is a win across the board.
Note that I ran these benchmarks with everything in RAM disk and using the
cpupower instructions that Nazir suggested.
So on Arm, the v3 COPY FROM patch is almost all upside, and the v4 COPY TO
patch is all upside. The same is almost true for Intel, but the CSV COPY
FROM regression, even from the V3 COPY FROM patch, is about 5%. The v4.2
COPY FROM patch always performs worse than the v3 COPY FROM patch in
worst-case scenarios.
Does it seem reasonable to stop performance testing the v4.2 COPY FROM
patch? Have we collected enough benchmark data to be confident that the v3
COPY FROM patch is the one we should be moving forward with?
Hope you are all having a great day,
-Manni
--
-- Manni Wood EDB: https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-12 20:42 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Mark Wong <markwkm@gmail.com>
2025-12-12 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-18 07:35 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-12-24 15:07 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-12-31 13:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-06 20:05 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-01-07 19:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-01-09 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-13 19:12 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Mark Wong <markwkm@gmail.com>
2026-01-14 00:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-01-17 21:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-01-20 20:48 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
@ 2026-01-22 18:22 ` KAZAR Ayoub <ma_kazar@esi.dz>
0 siblings, 0 replies; 178+ messages in thread
From: KAZAR Ayoub @ 2026-01-22 18:22 UTC (permalink / raw)
To: Manni Wood <manni.wood@enterprisedb.com>; +Cc: Mark Wong <markwkm@gmail.com>; Nazir Bilal Yavuz <byavuz81@gmail.com>; Nathan Bossart <nathandbossart@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
Hello,
On Tue, Jan 20, 2026 at 9:49 PM Manni Wood <manni.wood@enterprisedb.com>
wrote:
> Hello, all I have more benchmarks.
>
> These benchmarks are from a Raspberry Pi 5 that I bought. It has an Arm
> Cortex A76 processor.
>
> (I was so impressed with the stability of the results I got on my
> standalone Intel tower PC that I figured I needed a standalone Arm-based
> machine that was not a laptop and not a VM at a cloud service provider. The
> run-to-run results were indeed more stable, just like with my standalone
> tower PC.)
>
> COPY FROM
>
> master: (852558b9)
>
> text, no special: 9111
> text, 1/3 special: 10302
> csv, no special: 11147
> csv, 1/3 special: 13375
>
> v3
>
> text, no special: 7351 (19.3% speedup)
> text, 1/3 special: 10397 (0.9% regression)
> csv, no special: 7272 (34.7% speedup)
> csv, 1/3 special: 13472 (0.7% regression)
>
> v4.2
>
> text, no special: 7300 (19.6% speedup)
> text, 1/3 special: 10537 (2.3% regression)
> csv, no special: 7260 (34.8% speedup)
> csv, 1/3 special: 13881 (3.8% regression)
>
> COPY TO
>
> master: (852558b9)
>
> text, no special: 2446
> text, 1/3 special: 6988
> csv, no special: 2822
> csv, 1/3 special: 6967
>
> v4 (copy to)
>
> text, no special: 1533 (37.3% speedup)
> text, 1/3 special: 5949 (14.8% speedup)
> csv, no special: 1560 (44.7% speedup)
> csv, 1/3 special: 6006 (13.8% speedup)
>
> I find these results particularly exciting because with the COPY FROM v3
> patch, the worst-case scenarios are just under 1% regression. The v4 COPY
> TO patch is a win across the board.
>
> Note that I ran these benchmarks with everything in RAM disk and using the
> cpupower instructions that Nazir suggested.
>
> So on Arm, the v3 COPY FROM patch is almost all upside, and the v4 COPY TO
> patch is all upside. The same is almost true for Intel, but the CSV COPY
> FROM regression, even from the V3 COPY FROM patch, is about 5%. The v4.2
> COPY FROM patch always performs worse than the v3 COPY FROM patch in
> worst-case scenarios.
>
> Does it seem reasonable to stop performance testing the v4.2 COPY FROM
> patch? Have we collected enough benchmark data to be confident that the v3
> COPY FROM patch is the one we should be moving forward with?
>
For the case of v4.2 using the 1/3 specials benchmark, it will always take
the decision to not use SIMD after sampling and that 3%-4% regression is
the combination of the small overhead of counting special characters and
2-4 branches and its effect on the general layout, branch prediction,
pipeline ..etc, while i don't think it's more complex than v3 but this is
the only thing i can think of.
And since it assumes uniformity of special characters between lines so yes
IMHO v3 is generally better.
Regards,
Ayoub
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-12 20:42 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Mark Wong <markwkm@gmail.com>
2025-12-12 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-18 07:35 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-12-24 15:07 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-12-31 13:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-06 20:05 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-01-07 19:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-01-09 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-13 19:12 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Mark Wong <markwkm@gmail.com>
2026-01-14 00:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
@ 2026-01-30 22:05 ` Mark Wong <markwkm@gmail.com>
2026-02-02 20:45 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
1 sibling, 1 reply; 178+ messages in thread
From: Mark Wong @ 2026-01-30 22:05 UTC (permalink / raw)
To: Manni Wood <manni.wood@enterprisedb.com>; +Cc: Nazir Bilal Yavuz <byavuz81@gmail.com>; KAZAR Ayoub <ma_kazar@esi.dz>; Nathan Bossart <nathandbossart@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
On Tue, Jan 13, 2026 at 06:20:27PM -0600, Manni Wood wrote:
> On Tue, Jan 13, 2026 at 1:12 PM Mark Wong <markwkm@gmail.com> wrote:
>
> > On Fri, Jan 09, 2026 at 05:21:45PM +0300, Nazir Bilal Yavuz wrote:
> > > Were you able to understand why Mark's benchmark results are different
> > > from ours?
> >
> > Not yet... I had some guesses, which is why I suggested the processor
> > pinning
> > and using a ramdisk. But we haven't tried applying all of those to my
> > laptop,
> > which has 3 core types, or the POWER system, which may be interesting to
> > use a
> > ram disk on.
> >
> > I'm curious though, and admittedly haven't tried looking myself yet, about
> > how
> > the SIMD calls might look across different processor architectures. We'll
> > try
> > to get that on the POWER system soon...
> >
> > Regards,
> > Mark
>
> Hello!
>
> Nazir, I'm glad you are finding the benchmarks useful. I have more! :-)
>
> All of these benchmarks are all-in-RAM, because I do think that is the best
> way of getting closest to the theoretical best and worst case scenarios.
>
> My laptop:
>
> master: (852558b9)
>
> text, no special: 14996
> text, 1/3 special: 17270
> csv, no special: 18274
> csv, 1/3 special: 23852
>
> v3
>
> text, no special: 11282 (24.7% speedup)
> text, 1/3 special: 15748 (8.8% speedup) <-- I don't believe this but it's
> what I got
> csv, no special: 11571 (36.6% speedup)
> csv, 1/3 special: 19934 (16.4% speedup) <-- I don't believe this but it's
> what I got
>
> v4.2
>
> text, no special: 11139 (25.7% speedup)
> text, 1/3 special: 18900 (9.4% regression)
> csv, no special: 11490 (37.1% speedup)
> csv, 1/3 special: 26134 (9.5% regression)
>
> An AWS EC2 t2.2xlarge instance
>
> master: (852558b9)
>
> text, no special: 20677
> text, 1/3 special: 22660
> csv, no special: 24534
> csv, 1/3 special: 30999
>
> v3
>
> text, no special: 17534 (15.2% speedup)
> text, 1/3 special: 22816 (0.6% regression)
> csv, no special: 17664 (28.0% speedup)
> csv, 1/3 special: 29338 (5.3% speedup) <-- I don't believe this but it's
> what I got
>
> v4.2
>
> text, no special: 17459 (15.5% speedup)
> text, 1/3 special: 25051 (10.5% regression)
> csv, no special: 17574 (28.3% speedup)
> csv, 1/3 special: 32092 (3.5% regression)
>
> An AWS EC2 t4g.2xlarge instance (using ARM processor; first test of ARM
> processor!)
>
> master: (852558b9)
>
> text, no special: 22081
> text, 1/3 special: 25100
> csv, no special: 27296
> csv, 1/3 special: 32344
>
> v3
>
> text, no special: 17724 (19.7% speedup)
> text, 1/3 special: 27606 (9.9% regression) <-- yikes! We would want to test
> this more
> csv, no special: 17597 (35.5% speedup)
> csv, 1/3 special: 32597 (0.8% regression)
>
> v4.2
>
> text, no special: 17674 (20% speedup)
> text, 1/3 special: 25773 (2.6% regression) <-- this regression is less than
> for the v3 patch? Atypical...
> csv, no special: 17651 (35.3% speedup)
> csv, 1/3 special: 34055 (5.3% regression)
I'm still lagging behind a little I ran the v4.2 patches again applied to
71c11369 on the POWER system that I have access to, using Manni's copysimdperf
scripts to use a ramdisk and processor pinning.
text, no special: -2508 (30% speedup)
text, 1/3 special: -1753 (48% speedup)
csv, no special: 9264 (3% regression)
csv, 1/3 special: -4077 (0.3% speedup)
Using Manni's script makes me feel more confident about executing the tests in
the same way, so I don't know how concerning the difference in results are
compared to the other architectures.
> Yes, I think I agree with you that the everything-in-RAM benchmarks will
> make the regressions more pronounced, just like the everything-in-RAM
> benchmarks make the improvements more pronounced.
>
> I am not sure why the CSV regression, compared to the TXT regression (even
> for the v3 patch which has smaller regressions than the v4.2 patch) is
> usually worse. I probably should look over some flame graphs and see if I
> can find the place where the CSV-parsing code is so much slower. The CSV
> regression is actually a bit frustrating (at around 5%) because the TXT
> regression, at less than 1% (for the v3 patch) is so much easier to bare.
>
> Here are some copy-to benchmarks for the v4 patch that applies SIMD to the
> copy-to code.
>
> These were all-in-RAM tests.
>
> My laptop
>
> master: (852558b9)
>
> text, no special: 2948
> text, 1/3 special: 11258
> csv, no special: 6245
> csv, 1/3 special: 11258
>
> v4 (copy to)
>
> text, no special: 2126 (27.9% speedup)
> text, 1/3 special: 12080 (7.3% regression) <-- did not see such a big
> regression before
> csv, no special: 2432 (61.0% speedup)
> csv, 1/3 special: 12344 (4.0% regression) <-- did not see such a big
> regression before
>
> An AWS EC2 t2.2xlarge instance
>
> master: (852558b9)
>
> text, no special: 4647
> text, 1/3 special: 13865
> csv, no special: 5421
> csv, 1/3 special: 15284
>
> v4 (copy to)
>
> text, no special: 2460 (47.0% speedup)
> text, 1/3 special: 14023 (1.1% regression)
> csv, no special: 2667 (50.7% speedup)
> csv, 1/3 special: 15251 (0.2% speedup)
>
> An AWS EC2 t4g.2xlarge instance (using ARM processor; first test of ARM
> processor!)
>
> master: (852558b9)
>
> text, no special: 6951
> text, 1/3 special: 17857
> csv, no special: 7951
> csv, 1/3 special: 18504
>
> v4 (copy to)
>
> text, no special: 3372 (51.4% speedup)
> text, 1/3 special: 15713 (12.0% speedup)
> csv, no special: 3233 (59.3% speedup)
> csv, 1/3 special: 1622 (12.3% speedup)
>
> Once again, the v4 patch for copy-to seems like a clearer win, though, to
> be fair, there were regressions when running on my laptop. (I'm starting to
> think servers or desktops are better than laptops for testing these things,
> though maybe that's my bias: it just seems like the server results are
> always less surprising.)
>
> Hope you all continue to find these useful...
Regards,
Mark
--
Mark Wong <markwkm@gmail.com>
EDB https://enterprisedb.com
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-12 20:42 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Mark Wong <markwkm@gmail.com>
2025-12-12 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-18 07:35 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-12-24 15:07 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-12-31 13:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-06 20:05 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-01-07 19:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-01-09 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-13 19:12 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Mark Wong <markwkm@gmail.com>
2026-01-14 00:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-01-30 22:05 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Mark Wong <markwkm@gmail.com>
@ 2026-02-02 20:45 ` KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-03 10:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
0 siblings, 1 reply; 178+ messages in thread
From: KAZAR Ayoub @ 2026-02-02 20:45 UTC (permalink / raw)
To: Mark Wong <markwkm@gmail.com>; Neil Conway <neil.conway@gmail.com>; +Cc: Manni Wood <manni.wood@enterprisedb.com>; Nazir Bilal Yavuz <byavuz81@gmail.com>; Nathan Bossart <nathandbossart@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
Hello Mark,
On Fri, Jan 30, 2026 at 11:05 PM Mark Wong <markwkm@gmail.com> wrote:
> On Tue, Jan 13, 2026 at 06:20:27PM -0600, Manni Wood wrote:
> > On Tue, Jan 13, 2026 at 1:12 PM Mark Wong <markwkm@gmail.com> wrote:
> >
> > > On Fri, Jan 09, 2026 at 05:21:45PM +0300, Nazir Bilal Yavuz wrote:
> > > > Were you able to understand why Mark's benchmark results are
> different
> > > > from ours?
> > >
> > > Not yet... I had some guesses, which is why I suggested the processor
> > > pinning
> > > and using a ramdisk. But we haven't tried applying all of those to my
> > > laptop,
> > > which has 3 core types, or the POWER system, which may be interesting
> to
> > > use a
> > > ram disk on.
> > >
> > > I'm curious though, and admittedly haven't tried looking myself yet,
> about
> > > how
> > > the SIMD calls might look across different processor architectures.
> We'll
> > > try
> > > to get that on the POWER system soon...
> > >
> > > Regards,
> > > Mark
> >
> > Hello!
> >
> > Nazir, I'm glad you are finding the benchmarks useful. I have more! :-)
> >
> > All of these benchmarks are all-in-RAM, because I do think that is the
> best
> > way of getting closest to the theoretical best and worst case scenarios.
> >
> > My laptop:
> >
> > master: (852558b9)
> >
> > text, no special: 14996
> > text, 1/3 special: 17270
> > csv, no special: 18274
> > csv, 1/3 special: 23852
> >
> > v3
> >
> > text, no special: 11282 (24.7% speedup)
> > text, 1/3 special: 15748 (8.8% speedup) <-- I don't believe this but it's
> > what I got
> > csv, no special: 11571 (36.6% speedup)
> > csv, 1/3 special: 19934 (16.4% speedup) <-- I don't believe this but it's
> > what I got
> >
> > v4.2
> >
> > text, no special: 11139 (25.7% speedup)
> > text, 1/3 special: 18900 (9.4% regression)
> > csv, no special: 11490 (37.1% speedup)
> > csv, 1/3 special: 26134 (9.5% regression)
> >
> > An AWS EC2 t2.2xlarge instance
> >
> > master: (852558b9)
> >
> > text, no special: 20677
> > text, 1/3 special: 22660
> > csv, no special: 24534
> > csv, 1/3 special: 30999
> >
> > v3
> >
> > text, no special: 17534 (15.2% speedup)
> > text, 1/3 special: 22816 (0.6% regression)
> > csv, no special: 17664 (28.0% speedup)
> > csv, 1/3 special: 29338 (5.3% speedup) <-- I don't believe this but it's
> > what I got
> >
> > v4.2
> >
> > text, no special: 17459 (15.5% speedup)
> > text, 1/3 special: 25051 (10.5% regression)
> > csv, no special: 17574 (28.3% speedup)
> > csv, 1/3 special: 32092 (3.5% regression)
> >
> > An AWS EC2 t4g.2xlarge instance (using ARM processor; first test of ARM
> > processor!)
> >
> > master: (852558b9)
> >
> > text, no special: 22081
> > text, 1/3 special: 25100
> > csv, no special: 27296
> > csv, 1/3 special: 32344
> >
> > v3
> >
> > text, no special: 17724 (19.7% speedup)
> > text, 1/3 special: 27606 (9.9% regression) <-- yikes! We would want to
> test
> > this more
> > csv, no special: 17597 (35.5% speedup)
> > csv, 1/3 special: 32597 (0.8% regression)
> >
> > v4.2
> >
> > text, no special: 17674 (20% speedup)
> > text, 1/3 special: 25773 (2.6% regression) <-- this regression is less
> than
> > for the v3 patch? Atypical...
> > csv, no special: 17651 (35.3% speedup)
> > csv, 1/3 special: 34055 (5.3% regression)
>
> I'm still lagging behind a little I ran the v4.2 patches again applied to
> 71c11369 on the POWER system that I have access to, using Manni's
> copysimdperf
> scripts to use a ramdisk and processor pinning.
>
> text, no special: -2508 (30% speedup)
> text, 1/3 special: -1753 (48% speedup)
> csv, no special: 9264 (3% regression)
> csv, 1/3 special: -4077 (0.3% speedup)
>
> Thanks for the benchmark, I'm a bit suspicious about this because I find
it illogical or at least highly unexpected for a 1/3 specials workload to
perform better than no specials !
or csv no special regressing ! because it's expected to take the simd path
for the whole processing, so it's supposed to perform better than master
(at least ...).
I wonder what the results look like for COPY TO case on POWER. If you can
try, that case is at least even more theoretically predictable.
Regards,
Ayoub
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-12 20:42 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Mark Wong <markwkm@gmail.com>
2025-12-12 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-18 07:35 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-12-24 15:07 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-12-31 13:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-06 20:05 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-01-07 19:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-01-09 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-13 19:12 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Mark Wong <markwkm@gmail.com>
2026-01-14 00:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-01-30 22:05 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Mark Wong <markwkm@gmail.com>
2026-02-02 20:45 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
@ 2026-02-03 10:59 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-03 11:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
0 siblings, 1 reply; 178+ messages in thread
From: Nazir Bilal Yavuz @ 2026-02-03 10:59 UTC (permalink / raw)
To: KAZAR Ayoub <ma_kazar@esi.dz>; +Cc: Mark Wong <markwkm@gmail.com>; Neil Conway <neil.conway@gmail.com>; Manni Wood <manni.wood@enterprisedb.com>; Nathan Bossart <nathandbossart@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
Hi,
On Mon, 2 Feb 2026 at 23:45, KAZAR Ayoub <ma_kazar@esi.dz> wrote:
>
>>
> Thanks for the benchmark, I'm a bit suspicious about this because I find it illogical or at least highly unexpected for a 1/3 specials workload to perform better than no specials !
> or csv no special regressing ! because it's expected to take the simd path for the whole processing, so it's supposed to perform better than master (at least ...).
I agree with you. I think it is highly unexpected that 1/3 specials
cases perform better than no special cases.
> I wonder what the results look like for COPY TO case on POWER. If you can try, that case is at least even more theoretically predictable.
Yes, that would be helpful.
--
Regards,
Nazir Bilal Yavuz
Microsoft
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-12 20:42 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Mark Wong <markwkm@gmail.com>
2025-12-12 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-18 07:35 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-12-24 15:07 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-12-31 13:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-06 20:05 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-01-07 19:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-01-09 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-13 19:12 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Mark Wong <markwkm@gmail.com>
2026-01-14 00:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-01-30 22:05 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Mark Wong <markwkm@gmail.com>
2026-02-02 20:45 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-03 10:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
@ 2026-02-03 11:02 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-03 12:06 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
0 siblings, 1 reply; 178+ messages in thread
From: Nazir Bilal Yavuz @ 2026-02-03 11:02 UTC (permalink / raw)
To: KAZAR Ayoub <ma_kazar@esi.dz>; +Cc: Mark Wong <markwkm@gmail.com>; Neil Conway <neil.conway@gmail.com>; Manni Wood <manni.wood@enterprisedb.com>; Nathan Bossart <nathandbossart@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
Hi,
Thank you for all your help!
This thread has a couple of discussions and it can be hard to follow;
I think it would be better if we can separate these discussions. This
would make it easier to follow the topics. What I have in my mind is:
* There are four patches in this thread, three of them COPY FROM
related: v3, v4 and v5. One of them is COPY TO related. So, I guess it
would be better to move COPY TO discussion to another thread.
* Manni suggested to stop benchmarking v4.2 COPY FROM patch, I agree
with him. Then it would be easier to address reviews and do
benchmarking.
* Kazar applied Neil's review on top of v3 and benchmarked, v3 seems
to perform better but I think flushing line_buf needs more
benchmarking.
* I am planning to create a new version (v6) of the patch, which will
be basically v3 with one less branching (moving !in_quote check under
is_csv and else). So that it would be easy to review and follow the
thread for new reviewers.
What do you think about these? Do you think these make sense and / or
do you have any other suggestions?
--
Regards,
Nazir Bilal Yavuz
Microsoft
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-12 20:42 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Mark Wong <markwkm@gmail.com>
2025-12-12 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-18 07:35 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-12-24 15:07 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-12-31 13:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-06 20:05 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-01-07 19:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-01-09 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-13 19:12 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Mark Wong <markwkm@gmail.com>
2026-01-14 00:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-01-30 22:05 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Mark Wong <markwkm@gmail.com>
2026-02-02 20:45 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-03 10:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-03 11:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
@ 2026-02-03 12:06 ` KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-04 05:38 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
0 siblings, 1 reply; 178+ messages in thread
From: KAZAR Ayoub @ 2026-02-03 12:06 UTC (permalink / raw)
To: Nazir Bilal Yavuz <byavuz81@gmail.com>; +Cc: Mark Wong <markwkm@gmail.com>; Neil Conway <neil.conway@gmail.com>; Manni Wood <manni.wood@enterprisedb.com>; Nathan Bossart <nathandbossart@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
Hello,
On Tue, Feb 3, 2026, 12:02 PM Nazir Bilal Yavuz <byavuz81@gmail.com> wrote:
> Hi,
>
> * There are four patches in this thread, three of them COPY FROM
> related: v3, v4 and v5. One of them is COPY TO related. So, I guess it
> would be better to move COPY TO discussion to another thread.
>
If there's possible discussions concerning COPY TO, then yes because i
don't find many concerns about COPY TO.
> * I am planning to create a new version (v6) of the patch, which will
> be basically v3 with one less branching (moving !in_quote check under
> is_csv and else). So that it would be easy to review and follow the
> thread for new reviewers.
>
Oh cool, looking forward to this.
Regards,
Ayoub
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-12 20:42 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Mark Wong <markwkm@gmail.com>
2025-12-12 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-18 07:35 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-12-24 15:07 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-12-31 13:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-06 20:05 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-01-07 19:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-01-09 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-13 19:12 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Mark Wong <markwkm@gmail.com>
2026-01-14 00:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-01-30 22:05 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Mark Wong <markwkm@gmail.com>
2026-02-02 20:45 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-03 10:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-03 11:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-03 12:06 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
@ 2026-02-04 05:38 ` Manni Wood <manni.wood@enterprisedb.com>
2026-02-04 14:28 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
0 siblings, 1 reply; 178+ messages in thread
From: Manni Wood @ 2026-02-04 05:38 UTC (permalink / raw)
To: KAZAR Ayoub <ma_kazar@esi.dz>; +Cc: Nazir Bilal Yavuz <byavuz81@gmail.com>; Mark Wong <markwkm@gmail.com>; Neil Conway <neil.conway@gmail.com>; Nathan Bossart <nathandbossart@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
On Tue, Feb 3, 2026 at 6:06 AM KAZAR Ayoub <ma_kazar@esi.dz> wrote:
> Hello,
>
> On Tue, Feb 3, 2026, 12:02 PM Nazir Bilal Yavuz <byavuz81@gmail.com>
> wrote:
>
>> Hi,
>>
>> * There are four patches in this thread, three of them COPY FROM
>> related: v3, v4 and v5. One of them is COPY TO related. So, I guess it
>> would be better to move COPY TO discussion to another thread.
>>
> If there's possible discussions concerning COPY TO, then yes because i
> don't find many concerns about COPY TO.
>
>> * I am planning to create a new version (v6) of the patch, which will
>> be basically v3 with one less branching (moving !in_quote check under
>> is_csv and else). So that it would be easy to review and follow the
>> thread for new reviewers.
>>
> Oh cool, looking forward to this.
>
> Regards,
> Ayoub
>
Hello!
I have run some more tests.
x86 tower PC
commit 78bf28e3bf504db0eea5e3bcb3c43e9908108480 (HEAD -> master,
origin/master, origin/HEAD)
Text, no special characters 30468.4045ms
CSV, no special characters 35351.084ms
Text, with 1/3 escapes 32722.48625ms
CSV, with 1/3 quotes 44044.87575ms
x86 tower PC
commit 78bf28e3bf504db0eea5e3bcb3c43e9908108480 (HEAD -> master,
origin/master, origin/HEAD)
plus
0001-COPY-from-SIMD-v3-with-line_buf-periodic-refill.patch
Text, no special characters 23538.6895ms 22.7439379% improvement
CSV, no special characters 23475.94ms 33.59202224% improvement
Text, with 1/3 escapes 33530.8395ms -2.470329558% regression
CSV, with 1/3 quotes 45268.28275ms -2.777637533 regression
x86 tower PC
commit 78bf28e3bf504db0eea5e3bcb3c43e9908108480 (HEAD -> master,
origin/master, origin/HEAD)
plus
v5.1-0001-Simple-heuristic-for-SIMD-COPY-FROM.patch.patch
Text, no special characters 22728.42475ms 25.40329852% improvement
CSV, no special characters 22777.7805ms 35.56695319% improvement
Text, with 1/3 escapes 34542.75625ms -5.562749683% regression
CSV, with 1/3 quotes 45793.0095 -3.96898327 regression
arm raspberry pi 5
commit 78bf28e3bf504db0eea5e3bcb3c43e9908108480 (HEAD -> master,
postgres/master, postgres/HEAD)
Text, no special characters 9476.60875ms
CSV, no special characters 11132.6405ms
Text, with 1/3 escapes 10765.8125ms
CSV, with 1/3 quotes 14055.28925ms
arm raspberry pi 5
commit 78bf28e3bf504db0eea5e3bcb3c43e9908108480 (HEAD -> master,
postgres/master, postgres/HEAD)
with
0001-COPY-from-SIMD-v3-with-line_buf-periodic-refill.patch
Text, no special characters 7380.328ms 22.12057926% improvement
CSV, no special characters 7349.53475ms 33.98210649% improvement
Text, with 1/3 escapes 10350.6385ms 3.856411209% improvement
CSV, with 1/3 quotes 12407.22725ms 11.72556445% improvement
arm raspberry pi 5
commit 78bf28e3bf504db0eea5e3bcb3c43e9908108480 (HEAD -> master,
postgres/master, postgres/HEAD)
with
v5.1-0001-Simple-heuristic-for-SIMD-COPY-FROM.patch.patch
Text, no special characters 7379.0375ms 22.134197% improvement
CSV, no special characters 7411.73225ms 33.42341154% improvement
Text, with 1/3 escapes 11288.465ms -4.854742733% regression
CSV, with 1/3 quotes 15281.3355ms -8.723023968% regression
The 0001-COPY-from-SIMD-v3-with-line_buf-periodic-refill.patch seems nice!
On My x86 PC, it had the usual performance improvment of earlier patches,
but the regression seemed more similar for both text and csv inputs.
Unfortunately, the regression is about 2.5%, but maybe that is an
acceptable worst-case for an improvement of 22% for text inputs and 33% for
CSV inputs?
The 0001-COPY-from-SIMD-v3-with-line_buf-periodic-refill.patch looks even
better on my Raspberry Pi's arm processor: not only do we see a 22%
improvement for text and an almost 34% improvement for CSV, even the
worst-case scenarios show an almost 4% improvement for text and an 11.7%
improvement for CSV.
By comparison,
the v5.1-0001-Simple-heuristic-for-SIMD-COPY-FROM.patch.patch's worst-case
performance is poorer on both architectures.
I'd be curious to know if anyone else can reproduces these
numbers. 0001-COPY-from-SIMD-v3-with-line_buf-periodic-refill.patch seems
like a real winner.
Best,
-Manni
--
-- Manni Wood EDB: https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-12 20:42 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Mark Wong <markwkm@gmail.com>
2025-12-12 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-18 07:35 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-12-24 15:07 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-12-31 13:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-06 20:05 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-01-07 19:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-01-09 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-13 19:12 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Mark Wong <markwkm@gmail.com>
2026-01-14 00:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-01-30 22:05 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Mark Wong <markwkm@gmail.com>
2026-02-02 20:45 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-03 10:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-03 11:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-03 12:06 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-04 05:38 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
@ 2026-02-04 14:28 ` KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-04 15:07 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
0 siblings, 1 reply; 178+ messages in thread
From: KAZAR Ayoub @ 2026-02-04 14:28 UTC (permalink / raw)
To: Manni Wood <manni.wood@enterprisedb.com>; +Cc: Nazir Bilal Yavuz <byavuz81@gmail.com>; Mark Wong <markwkm@gmail.com>; Neil Conway <neil.conway@gmail.com>; Nathan Bossart <nathandbossart@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
Hello,
On Wed, Feb 4, 2026, 6:38 AM Manni Wood <manni.wood@enterprisedb.com> wrote:
> The 0001-COPY-from-SIMD-v3-with-line_buf-periodic-refill.patch seems nice!
> On My x86 PC, it had the usual performance improvment of earlier patches,
> but the regression seemed more similar for both text and csv inputs.
> Unfortunately, the regression is about 2.5%, but maybe that is an
> acceptable worst-case for an improvement of 22% for text inputs and 33% for
> CSV inputs?
>
> The 0001-COPY-from-SIMD-v3-with-line_buf-periodic-refill.patch looks even
> better on my Raspberry Pi's arm processor: not only do we see a 22%
> improvement for text and an almost 34% improvement for CSV, even the
> worst-case scenarios show an almost 4% improvement for text and an 11.7%
> improvement for CSV.
>
> By comparison,
> the v5.1-0001-Simple-heuristic-for-SIMD-COPY-FROM.patch.patch's worst-case
> performance is poorer on both architectures.
>
> I'd be curious to know if anyone else can reproduces these
> numbers. 0001-COPY-from-SIMD-v3-with-line_buf-periodic-refill.patch seems
> like a real winner.
>
Thanks for the benchmark Manni, i suppose this is the same threshold as
patch has (4096 bytes), have you tried any bigger values for the threshold
?
Because i'm still expecting less l1d cache misses and execution times the
more we increase the threshold (relatively to l1d cache size per core).
As per my previous not-so-stable numbers 28KB wasn't too bad.
Regards,
Ayoub
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-12 20:42 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Mark Wong <markwkm@gmail.com>
2025-12-12 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-18 07:35 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-12-24 15:07 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-12-31 13:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-06 20:05 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-01-07 19:13 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-01-09 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-13 19:12 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Mark Wong <markwkm@gmail.com>
2026-01-14 00:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-01-30 22:05 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Mark Wong <markwkm@gmail.com>
2026-02-02 20:45 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-03 10:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-03 11:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-03 12:06 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-04 05:38 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2026-02-04 14:28 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
@ 2026-02-04 15:07 ` Manni Wood <manni.wood@enterprisedb.com>
0 siblings, 0 replies; 178+ messages in thread
From: Manni Wood @ 2026-02-04 15:07 UTC (permalink / raw)
To: KAZAR Ayoub <ma_kazar@esi.dz>; +Cc: Nazir Bilal Yavuz <byavuz81@gmail.com>; Mark Wong <markwkm@gmail.com>; Neil Conway <neil.conway@gmail.com>; Nathan Bossart <nathandbossart@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
On Wed, Feb 4, 2026 at 8:29 AM KAZAR Ayoub <ma_kazar@esi.dz> wrote:
> Hello,
>
> On Wed, Feb 4, 2026, 6:38 AM Manni Wood <manni.wood@enterprisedb.com>
> wrote:
>
>> The 0001-COPY-from-SIMD-v3-with-line_buf-periodic-refill.patch seems
>> nice! On My x86 PC, it had the usual performance improvment of earlier
>> patches, but the regression seemed more similar for both text and csv
>> inputs. Unfortunately, the regression is about 2.5%, but maybe that is an
>> acceptable worst-case for an improvement of 22% for text inputs and 33% for
>> CSV inputs?
>>
>> The 0001-COPY-from-SIMD-v3-with-line_buf-periodic-refill.patch looks even
>> better on my Raspberry Pi's arm processor: not only do we see a 22%
>> improvement for text and an almost 34% improvement for CSV, even the
>> worst-case scenarios show an almost 4% improvement for text and an 11.7%
>> improvement for CSV.
>>
>> By comparison,
>> the v5.1-0001-Simple-heuristic-for-SIMD-COPY-FROM.patch.patch's worst-case
>> performance is poorer on both architectures.
>>
>> I'd be curious to know if anyone else can reproduces these
>> numbers. 0001-COPY-from-SIMD-v3-with-line_buf-periodic-refill.patch seems
>> like a real winner.
>>
> Thanks for the benchmark Manni, i suppose this is the same threshold as
> patch has (4096 bytes), have you tried any bigger values for the threshold
> ?
> Because i'm still expecting less l1d cache misses and execution times the
> more we increase the threshold (relatively to l1d cache size per core).
> As per my previous not-so-stable numbers 28KB wasn't too bad.
>
>
> Regards,
> Ayoub
>
Ah, thanks for the prod, Ayoub. You are correct: The results in my previous
e-mail for the 0001-COPY-from-SIMD-v3-with-line_buf-periodic-refill.patch
patch are with LINE_BUF_FLUSH_AFTER set to its default of 4096. I will try
to measure what happens for larger LINE_BUF_FLUSH_AFTER values, hopefully
some time this week.
Best,
-Manni
--
-- Manni Wood EDB: https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-12 20:42 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Mark Wong <markwkm@gmail.com>
2025-12-12 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-18 07:35 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-12-24 15:07 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-12-31 13:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-06 20:05 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
@ 2026-01-09 14:20 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
1 sibling, 0 replies; 178+ messages in thread
From: Nazir Bilal Yavuz @ 2026-01-09 14:20 UTC (permalink / raw)
To: Manni Wood <manni.wood@enterprisedb.com>; +Cc: KAZAR Ayoub <ma_kazar@esi.dz>; Mark Wong <markwkm@gmail.com>; Nathan Bossart <nathandbossart@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
Hi,
Firstly, thank you for all of the benchmarks!
On Tue, 6 Jan 2026 at 23:05, Manni Wood <manni.wood@enterprisedb.com> wrote:
>
> Hello, Nazir, I tried your suggested cpupower commands as well as disabling turbo, and my results are indeed more uniform. (see attached screenshot of my spreadsheet).
I am glad that it helped!
> This time, I ran the tests on my Tower PC instead of on my laptop.
>
> I also followed Mark Wong's advice and used the taskset command to pin my postgres postmaster (and all of its children) to a single cpu core.
That is nice advice, I should apply this. Thank you for sharing it.
> So when I start postgres, I do this to pin it to core 27:
>
> ${PGHOME}/bin/pg_ctl -D ${PGHOME}/data -l ${PGHOME}/logfile.txt start
> PGPID=$(head -1 ${PGHOME}/data/postmaster.pid)
> taskset --cpu-list -p 27 ${PGPID}
>
>
> My results seem similar to yours:
It is nice to see that we get similar results.
> master: Nazir 85ddcc2f4c | Manni 877ae5db
>
> text, no special: 102294 | 302651
> text, 1/3 special: 108946 | 326208
> csv, no special: 121831 | 348930
> csv, 1/3 special: 140063 | 439786
>
> v3
>
> text, no special: 88890 (13.1% speedup) | 227874 (24.7% speedup)
> text, 1/3 special: 110463 (1.4% regression) | 322637 (1.1% speedup)
> csv, no special: 89781 (26.3% speedup) | 226525 (35.1% speedup)
> csv, 1/3 special: 147094 (5.0% regression) | 461501 (4.9% regression)
>
> v4.2
>
> text, no special: 87785 (14.2% speedup) | 225702 (25.4% speedup)
> text, 1/3 special: 127008 (16.6% regression) | 343480 (5.3% regression)
> csv, no special: 88093 (27.7% speedup) | 226633 (35.0% speedup)
> csv, 1/3 special: 164487 (17.4% regression) | 510954 (16.2% regression)
>
> It would seem that both your results and mine show a more serious worst-case regression for the v4.2 patches than for the v3 patches. It seems also that the speedups for v4.2 and v3 are similar.
Yes, you are right. Also, the regression on the CSV is worse than
TEXT, do you have any idea why?
--
Regards,
Nazir Bilal Yavuz
Microsoft
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 11:50 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Bilal Yavuz <byavuz81@gmail.com>
2025-12-12 20:42 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Mark Wong <markwkm@gmail.com>
2025-12-12 23:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-12-18 07:35 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-12-24 15:07 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-12-31 13:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
@ 2026-01-17 20:46 ` KAZAR Ayoub <ma_kazar@esi.dz>
1 sibling, 0 replies; 178+ messages in thread
From: KAZAR Ayoub @ 2026-01-17 20:46 UTC (permalink / raw)
To: Nazir Bilal Yavuz <byavuz81@gmail.com>; +Cc: Manni Wood <manni.wood@enterprisedb.com>; Mark Wong <markwkm@gmail.com>; Nathan Bossart <nathandbossart@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
Hello,
On Wed, Dec 31, 2025 at 2:04 PM Nazir Bilal Yavuz <byavuz81@gmail.com>
wrote:
> Hi,
>
> On Wed, 24 Dec 2025 at 18:08, KAZAR Ayoub <ma_kazar@esi.dz> wrote:
> >
> > Hello,
> > Following the same path of optimizing COPY FROM using SIMD, i found that
> COPY TO can also benefit from this.
> >
> > I attached a small patch that uses SIMD to skip data and advance as far
> as the first special character is found, then fallback to scalar processing
> for that character and re-enter the SIMD path again...
> > There's two ways to do this:
> > 1) Essentially we do SIMD until we find a special character, then
> continue scalar path without re-entering SIMD again.
> > - This gives from 10% to 30% speedups depending on the weight of special
> characters in the attribute, we don't lose anything here since it advances
> with SIMD until it can't (using the previous scripts: 1/3, 2/3 specials
> chars).
> >
> > 2) Do SIMD path, then use scalar path when we hit a special character,
> keep re-entering the SIMD path each time.
> > - This is equivalent to the COPY FROM story, we'll need to find the same
> heuristic to use for both COPY FROM/TO to reduce the regressions (same
> regressions: around from 20% to 30% with 1/3, 2/3 specials chars).
> >
> > Something else to note is that the scalar path for COPY TO isn't as
> heavy as the state machine in COPY FROM.
> >
> > So if we find the sweet spot for the heuristic, doing the same for COPY
> TO will be trivial and always beneficial.
> > Attached is 0004 which is option 1 (SIMD without re-entering), 0005 is
> the second one.
>
> Patches look correct to me. I think we could move these SIMD code
> portions into a shared function to remove duplication, although that
> might have a performance impact. I have not benchmarked these patches
> yet.
>
Definitely yes.
>
> Another consideration is that these patches might need their own
> thread, though I am not completely sure about this yet.
>
I thought maybe since it uses the same infrastructure, it needs/does the
same ideas and it's an easier problem than COPY FROM so this might be
interesting to be kept/committed together.
Regards,
Ayoub Kazar
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nathan Bossart <nathandbossart@gmail.com>
@ 2025-11-18 20:42 ` KAZAR Ayoub <ma_kazar@esi.dz>
1 sibling, 0 replies; 178+ messages in thread
From: KAZAR Ayoub @ 2025-11-18 20:42 UTC (permalink / raw)
To: Nathan Bossart <nathandbossart@gmail.com>; +Cc: Manni Wood <manni.wood@enterprisedb.com>; Andrew Dunstan <andrew@dunslane.net>; Nazir Bilal Yavuz <byavuz81@gmail.com>; Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
On Mon, Nov 17, 2025, 11:16 PM Nathan Bossart <nathandbossart@gmail.com>
wrote:
> (assuming there is a desire to
> continue with it)?
I'm hoping to start spending more time on it soon.
>
Somethings worth noting for future reference (so someone else wouldn't
waste time thinking about it), previously I tried extra several micro
optimizations inside and around CopyReadLineText:
SIMD alignment*:* Forcing 16-byte aligned buffers so we could use aligned
memory instructions (_mm_load_si128 vs _mm_loadu_si128) provided no
measurable benefit on modern CPUs (there's definitely a thread somewhere
talking about it that i didn't encounter yet). This likely explains why
simd.h exclusively uses unaligned load intrinsics the performance
difference has become negligible since Nehalem processors.
Memory prefetching: Explicit prefetch instructions for the COPY buffer
pipeline (copy_raw_buf, input buffers, etc.) either showed no improvement
or slight regression. Multiple chunks are already within a cache line,
other buffers are too far to prefetch and the next part of the buffer is
easily prefetched, nothing special, so it turns out to be not worth having
more uops.
Instruction-level parallelism: Spreading too many independent vector
operations to increase ILP eventually degrades performance, likely due to
backend saturation observed through perf (execution port and execution
units contention most likely ?)
.....
This simply suggests that further optimization work should focus on the
pipeline as a whole for large benefits (parallel copy[0], maybe ?).
[0]
https://www.postgresql.org/message-id/CAA4eK1+kpddvvLxWm4BuG_AhVvYz8mKAEa7osxp_X0d4ZEiV=g@mail.gmail...
--
Regards,
Ayoub Kazar
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` Re: Speed up COPY FROM text/CSV parsing using SIMD KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
@ 2025-08-21 19:36 ` KAZAR Ayoub <ma_kazar@esi.dz>
1 sibling, 0 replies; 178+ messages in thread
From: KAZAR Ayoub @ 2025-08-21 19:36 UTC (permalink / raw)
To: Nazir Bilal Yavuz <byavuz81@gmail.com>; +Cc: Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
> On Thu, 14 Aug 2025 at 18:00, KAZAR Ayoub <ma_kazar@esi.dz> wrote:
> >> Thanks for running that benchmark! Would you mind sharing a reproducer
> >> for the regression you observed?
> >
> > Of course, I attached the sql to generate the text and csv test files.
> > If having a 1/3 of line length of special characters can be an
> exaggeration, something lower might still reproduce some regressions of
> course for the same idea.
>
> Thank you so much!
>
> I am able to reproduce the regression you mentioned but both
> regressions are %20 on my end. I found that (by experimenting) SIMD
> causes a regression if it advances less than 5 characters.
>
> So, I implemented a small heuristic. It works like that:
>
> - If advance < 5 -> insert a sleep penalty (n cycles).
> - Each time advance < 5, n is doubled.
> - Each time advance ≥ 5, n is halved.
>
> I am sharing a POC patch to show heuristic, it can be applied on top
> of v1-0001. Heuristic version has the same performance improvements
> with the v1-0001 but the regression is %5 instead of %20 compared to
> the master.
>
> --
> Regards,
> Nazir Bilal Yavuz
> Microsoft
Yes this is good, i'm also getting about 5% regression only now.
Regards,
Ayoub Kazar
^ permalink raw reply [nested|flat] 178+ messages in thread
* Re: Speed up COPY FROM text/CSV parsing using SIMD
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Re: Speed up COPY FROM text/CSV parsing using SIMD Nazir Bilal Yavuz <byavuz81@gmail.com>
@ 2025-08-19 09:09 ` Ants Aasma <ants.aasma@cybertec.at>
1 sibling, 0 replies; 178+ messages in thread
From: Ants Aasma @ 2025-08-19 09:09 UTC (permalink / raw)
To: Nazir Bilal Yavuz <byavuz81@gmail.com>; +Cc: Shinya Kato <shinya11.kato@gmail.com>; pgsql-hackers
On Thu, 7 Aug 2025 at 14:15, Nazir Bilal Yavuz <byavuz81@gmail.com> wrote:
> I have a couple of ideas that I was working on:
> ---
>
> + * However, SIMD optimization cannot be applied in the following cases:
> + * - Inside quoted fields, where escape sequences and closing quotes
> + * require sequential processing to handle correctly.
>
> I think you can continue SIMD inside quoted fields. Only important
> thing is you need to set last_was_esc to false when SIMD skipped the
> chunk.
There is a trick with doing carryless multiplication with -1 that can
be used to SIMD process transitions between quoted/not-quoted. [1]
This is able to convert a bitmask of unescaped quote character
positions to a quote mask in a single operation. I last looked at it 5
years ago, but I remember coming to the conclusion that it would work
for implementing PostgreSQL's interpretation of CSV.
[1] https://github.com/geofflangdale/simdcsv/blob/master/src/main.cpp#L76
--
Ants
^ permalink raw reply [nested|flat] 178+ messages in thread
end of thread, other threads:[~2026-09-09 21:07 UTC | newest]
Thread overview: 178+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2025-08-07 01:48 Speed up COPY FROM text/CSV parsing using SIMD Shinya Kato <shinya11.kato@gmail.com>
2025-08-07 11:15 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-11 08:52 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-12 07:25 ` Shinya Kato <shinya11.kato@gmail.com>
2025-08-13 06:21 ` Shinya Kato <shinya11.kato@gmail.com>
2025-08-14 02:24 ` KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-14 10:29 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-14 14:59 ` KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 12:33 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-19 14:14 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-08-21 15:47 ` Andrew Dunstan <andrew@dunslane.net>
2025-10-16 14:29 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-18 18:46 ` KAZAR Ayoub <ma_kazar@esi.dz>
2025-10-18 20:01 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 06:17 ` KAZAR Ayoub <ma_kazar@esi.dz>
2025-10-21 06:44 ` KAZAR Ayoub <ma_kazar@esi.dz>
2025-10-21 18:55 ` Nathan Bossart <nathandbossart@gmail.com>
2025-10-18 20:01 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-20 14:02 ` Andrew Dunstan <andrew@dunslane.net>
2025-10-20 17:04 ` Nathan Bossart <nathandbossart@gmail.com>
2025-10-20 20:31 ` Andrew Dunstan <andrew@dunslane.net>
2025-10-20 21:09 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-21 18:40 ` Nathan Bossart <nathandbossart@gmail.com>
2025-10-22 12:33 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-10-22 19:24 ` Nathan Bossart <nathandbossart@gmail.com>
2025-10-29 22:22 ` Andrew Dunstan <andrew@dunslane.net>
2025-11-11 22:23 ` Manni Wood <manni.wood@enterprisedb.com>
2025-11-12 14:44 ` KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-13 02:40 ` Manni Wood <manni.wood@enterprisedb.com>
2025-11-17 22:16 ` Nathan Bossart <nathandbossart@gmail.com>
2025-11-17 22:52 ` Shinya Kato <shinya11.kato@gmail.com>
2025-11-18 08:04 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-18 14:01 ` Andrew Dunstan <andrew@dunslane.net>
2025-11-18 14:20 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-19 21:01 ` Nathan Bossart <nathandbossart@gmail.com>
2025-11-20 12:55 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-11-21 14:48 ` Andrew Dunstan <andrew@dunslane.net>
2025-11-24 21:59 ` Nathan Bossart <nathandbossart@gmail.com>
2025-11-26 00:09 ` Manni Wood <manni.wood@enterprisedb.com>
2025-11-26 11:50 ` KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-26 14:21 ` Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 01:39 ` Manni Wood <manni.wood@enterprisedb.com>
2025-12-06 07:55 ` Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 13:40 ` Bilal Yavuz <byavuz81@gmail.com>
2025-12-09 22:13 ` Manni Wood <manni.wood@enterprisedb.com>
2025-12-10 11:59 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-21 20:49 ` Neil Conway <neil.conway@gmail.com>
2026-01-22 19:32 ` KAZAR Ayoub <ma_kazar@esi.dz>
2026-01-31 16:20 ` KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-02 14:17 ` Manni Wood <manni.wood@enterprisedb.com>
2026-02-06 13:51 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 18:11 ` KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 21:29 ` Nathan Bossart <nathandbossart@gmail.com>
2026-02-06 22:19 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-06 22:36 ` KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-06 22:47 ` Nathan Bossart <nathandbossart@gmail.com>
2026-02-11 13:27 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-11 22:39 ` Nathan Bossart <nathandbossart@gmail.com>
2026-02-13 11:45 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-13 23:09 ` Nathan Bossart <nathandbossart@gmail.com>
2026-02-14 03:34 ` Manni Wood <manni.wood@enterprisedb.com>
2026-02-16 17:04 ` Nathan Bossart <nathandbossart@gmail.com>
2026-02-16 18:15 ` Nathan Bossart <nathandbossart@gmail.com>
2026-02-17 05:01 ` Manni Wood <manni.wood@enterprisedb.com>
2026-02-17 22:29 ` Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 13:38 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-18 21:26 ` Nathan Bossart <nathandbossart@gmail.com>
2026-02-18 21:54 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 04:02 ` Manni Wood <manni.wood@enterprisedb.com>
2026-02-19 09:01 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-19 22:37 ` KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-20 00:09 ` Manni Wood <manni.wood@enterprisedb.com>
2026-02-20 09:50 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-20 18:15 ` Nathan Bossart <nathandbossart@gmail.com>
2026-02-23 09:10 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 04:44 ` Manni Wood <manni.wood@enterprisedb.com>
2026-02-24 13:57 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-24 15:07 ` KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-24 17:48 ` Nathan Bossart <nathandbossart@gmail.com>
2026-02-25 04:06 ` Manni Wood <manni.wood@enterprisedb.com>
2026-02-25 14:24 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-26 12:19 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-26 14:31 ` KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-26 14:36 ` Manni Wood <manni.wood@enterprisedb.com>
2026-02-26 15:32 ` Manni Wood <manni.wood@enterprisedb.com>
2026-02-26 15:51 ` KAZAR Ayoub <ma_kazar@esi.dz>
2026-03-02 19:55 ` Nathan Bossart <nathandbossart@gmail.com>
2026-03-04 15:15 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-05 21:25 ` Andrew Dunstan <andrew@dunslane.net>
2026-03-06 16:59 ` Manni Wood <manni.wood@enterprisedb.com>
2026-03-06 17:39 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-06 18:13 ` Manni Wood <manni.wood@enterprisedb.com>
2026-03-06 18:55 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-06 21:25 ` Manni Wood <manni.wood@enterprisedb.com>
2026-03-06 23:13 ` Nathan Bossart <nathandbossart@gmail.com>
2026-03-06 23:31 ` KAZAR Ayoub <ma_kazar@esi.dz>
2026-03-08 10:31 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-08 19:45 ` Manni Wood <manni.wood@enterprisedb.com>
2026-03-09 08:10 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-09 13:31 ` Manni Wood <manni.wood@enterprisedb.com>
2026-03-09 13:43 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-09 18:25 ` Nathan Bossart <nathandbossart@gmail.com>
2026-03-10 02:30 ` Manni Wood <manni.wood@enterprisedb.com>
2026-03-10 11:42 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-10 12:35 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-10 17:10 ` Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 11:36 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 12:19 ` KAZAR Ayoub <ma_kazar@esi.dz>
2026-03-11 13:10 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 13:23 ` KAZAR Ayoub <ma_kazar@esi.dz>
2026-03-11 18:09 ` Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 18:49 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 19:02 ` Nathan Bossart <nathandbossart@gmail.com>
2026-03-11 19:22 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-11 20:42 ` Nathan Bossart <nathandbossart@gmail.com>
2026-03-12 10:59 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-12 17:37 ` Nathan Bossart <nathandbossart@gmail.com>
2026-03-13 02:39 ` Manni Wood <manni.wood@enterprisedb.com>
2026-03-13 15:00 ` Nathan Bossart <nathandbossart@gmail.com>
2026-03-13 11:57 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-13 13:34 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-13 14:05 ` Nathan Bossart <nathandbossart@gmail.com>
2026-03-13 15:29 ` Greg Burd <greg@burd.me>
2026-03-13 16:14 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-13 16:16 ` Nathan Bossart <nathandbossart@gmail.com>
2026-03-13 17:22 ` Greg Burd <greg@burd.me>
2026-03-13 17:21 ` Greg Burd <greg@burd.me>
2026-03-13 17:33 ` Nathan Bossart <nathandbossart@gmail.com>
2026-08-31 14:24 ` Nathan Bossart <nathandbossart@gmail.com>
2026-08-31 17:48 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-09-01 04:41 ` Greg Burd <greg@burd.me>
2026-09-01 08:17 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-09-01 19:23 ` Nathan Bossart <nathandbossart@gmail.com>
2026-09-07 16:29 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-09-08 10:11 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-09-08 15:08 ` Greg Burd <greg@burd.me>
2026-09-08 15:36 ` Nathan Bossart <nathandbossart@gmail.com>
2026-09-08 15:58 ` Greg Burd <greg@burd.me>
2026-09-08 20:58 ` Nathan Bossart <nathandbossart@gmail.com>
2026-09-09 10:12 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-09-09 11:14 ` Greg Burd <greg@burd.me>
2026-09-09 21:07 ` Nathan Bossart <nathandbossart@gmail.com>
2026-09-08 15:24 ` Nathan Bossart <nathandbossart@gmail.com>
2026-09-09 10:00 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-13 15:58 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-03-13 16:08 ` Nathan Bossart <nathandbossart@gmail.com>
2026-03-13 16:13 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-20 10:01 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-12-12 20:42 ` Mark Wong <markwkm@gmail.com>
2025-12-12 23:09 ` Manni Wood <manni.wood@enterprisedb.com>
2025-12-18 07:35 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2025-12-24 15:07 ` KAZAR Ayoub <ma_kazar@esi.dz>
2025-12-29 17:03 ` Manni Wood <manni.wood@enterprisedb.com>
2025-12-31 13:04 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-06 20:05 ` Manni Wood <manni.wood@enterprisedb.com>
2026-01-07 19:13 ` Manni Wood <manni.wood@enterprisedb.com>
2026-01-08 20:49 ` Manni Wood <manni.wood@enterprisedb.com>
2026-01-09 00:01 ` Manni Wood <manni.wood@enterprisedb.com>
2026-01-09 14:22 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-09 14:21 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-13 19:12 ` Mark Wong <markwkm@gmail.com>
2026-01-14 00:20 ` Manni Wood <manni.wood@enterprisedb.com>
2026-01-17 21:24 ` KAZAR Ayoub <ma_kazar@esi.dz>
2026-01-20 20:48 ` Manni Wood <manni.wood@enterprisedb.com>
2026-01-22 18:22 ` KAZAR Ayoub <ma_kazar@esi.dz>
2026-01-30 22:05 ` Mark Wong <markwkm@gmail.com>
2026-02-02 20:45 ` KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-03 10:59 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-03 11:02 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-02-03 12:06 ` KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-04 05:38 ` Manni Wood <manni.wood@enterprisedb.com>
2026-02-04 14:28 ` KAZAR Ayoub <ma_kazar@esi.dz>
2026-02-04 15:07 ` Manni Wood <manni.wood@enterprisedb.com>
2026-01-09 14:20 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-01-17 20:46 ` KAZAR Ayoub <ma_kazar@esi.dz>
2025-11-18 20:42 ` KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-21 19:36 ` KAZAR Ayoub <ma_kazar@esi.dz>
2025-08-19 09:09 ` Ants Aasma <ants.aasma@cybertec.at>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox