public inbox for [email protected]
help / color / mirror / Atom feed[PATCH 1/1] Fix a corner-case in COPY FROM backslash processing.
48+ messages / 5 participants
[nested] [flat]
* [PATCH 1/1] Fix a corner-case in COPY FROM backslash processing.
@ 2021-02-03 12:00 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 48+ messages in thread
From: Heikki Linnakangas @ 2021-02-03 12:00 UTC (permalink / raw)
If a multi-byte character is escaped with a backslash in TEXT mode input,
we didn't always treat the escape correctly. If:
- a multi-byte character is escaped with a backslash, and
- the second byte of the character is 0x5C, i.e. the ASCII code of a
backslash (\), and
- the next character is a dot (.), then
copyreadline we would incorrectly interpret the sequence as an end-of-copy
marker (\.). this can only happen in encodings that can "embed" ascii
characters as the second byte. one example of such sequence is
'\x5ca45c2e666f6f' in Big5 encoding. If you put that in a file, and
load it with COPY FROM, you'd incorrectly get an "end-of-copy marker
corrupt" error.
---
src/backend/commands/copyfromparse.c | 13 +++++++++++--
1 file changed, 11 insertions(+), 2 deletions(-)
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index 798e18e013..c20ec482db 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -1084,7 +1084,7 @@ CopyReadLineText(CopyFromState cstate)
break;
}
else if (!cstate->opts.csv_mode)
-
+ {
/*
* If we are here, it means we found a backslash followed by
* something other than a period. In non-CSV mode, anything
@@ -1095,8 +1095,17 @@ CopyReadLineText(CopyFromState cstate)
* backslashes are not special, so we want to process the
* character after the backslash just like a normal character,
* so we don't increment in those cases.
+ *
+ * In principle, we would need to use pg_encoding_mblen() to
+ * skip over the character in some encodings, like at the end
+ * of the loop. But if it's a multi-byte character, it cannot
+ * have any special meaning and skipping isn't necessary for
+ * correctness. We can fall through to process it normally
+ * instead.
*/
- raw_buf_ptr++;
+ if (!IS_HIGHBIT_SET(c2))
+ raw_buf_ptr++;
+ }
}
/*
--
2.30.0
--------------97F3138F3612F1A4F9200D93--
^ permalink raw reply [nested|flat] 48+ messages in thread
* [PATCH 1/1] Fix a corner-case in COPY FROM backslash processing.
@ 2021-02-03 12:00 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 48+ messages in thread
From: Heikki Linnakangas @ 2021-02-03 12:00 UTC (permalink / raw)
If a multi-byte character is escaped with a backslash in TEXT mode input,
we didn't always treat the escape correctly. If:
- a multi-byte character is escaped with a backslash, and
- the second byte of the character is 0x5C, i.e. the ASCII code of a
backslash (\), and
- the next character is a dot (.), then
copyreadline we would incorrectly interpret the sequence as an end-of-copy
marker (\.). this can only happen in encodings that can "embed" ascii
characters as the second byte. one example of such sequence is
'\x5ca45c2e666f6f' in Big5 encoding. If you put that in a file, and
load it with COPY FROM, you'd incorrectly get an "end-of-copy marker
corrupt" error.
---
src/backend/commands/copyfromparse.c | 13 +++++++++++--
1 file changed, 11 insertions(+), 2 deletions(-)
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index 798e18e013..c20ec482db 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -1084,7 +1084,7 @@ CopyReadLineText(CopyFromState cstate)
break;
}
else if (!cstate->opts.csv_mode)
-
+ {
/*
* If we are here, it means we found a backslash followed by
* something other than a period. In non-CSV mode, anything
@@ -1095,8 +1095,17 @@ CopyReadLineText(CopyFromState cstate)
* backslashes are not special, so we want to process the
* character after the backslash just like a normal character,
* so we don't increment in those cases.
+ *
+ * In principle, we would need to use pg_encoding_mblen() to
+ * skip over the character in some encodings, like at the end
+ * of the loop. But if it's a multi-byte character, it cannot
+ * have any special meaning and skipping isn't necessary for
+ * correctness. We can fall through to process it normally
+ * instead.
*/
- raw_buf_ptr++;
+ if (!IS_HIGHBIT_SET(c2))
+ raw_buf_ptr++;
+ }
}
/*
--
2.30.0
--------------97F3138F3612F1A4F9200D93--
^ permalink raw reply [nested|flat] 48+ messages in thread
* [PATCH 1/1] Fix a corner-case in COPY FROM backslash processing.
@ 2021-02-03 12:00 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 48+ messages in thread
From: Heikki Linnakangas @ 2021-02-03 12:00 UTC (permalink / raw)
If a multi-byte character is escaped with a backslash in TEXT mode input,
we didn't always treat the escape correctly. If:
- a multi-byte character is escaped with a backslash, and
- the second byte of the character is 0x5C, i.e. the ASCII code of a
backslash (\), and
- the next character is a dot (.), then
copyreadline we would incorrectly interpret the sequence as an end-of-copy
marker (\.). this can only happen in encodings that can "embed" ascii
characters as the second byte. one example of such sequence is
'\x5ca45c2e666f6f' in Big5 encoding. If you put that in a file, and
load it with COPY FROM, you'd incorrectly get an "end-of-copy marker
corrupt" error.
---
src/backend/commands/copyfromparse.c | 13 +++++++++++--
1 file changed, 11 insertions(+), 2 deletions(-)
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index 798e18e013..c20ec482db 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -1084,7 +1084,7 @@ CopyReadLineText(CopyFromState cstate)
break;
}
else if (!cstate->opts.csv_mode)
-
+ {
/*
* If we are here, it means we found a backslash followed by
* something other than a period. In non-CSV mode, anything
@@ -1095,8 +1095,17 @@ CopyReadLineText(CopyFromState cstate)
* backslashes are not special, so we want to process the
* character after the backslash just like a normal character,
* so we don't increment in those cases.
+ *
+ * In principle, we would need to use pg_encoding_mblen() to
+ * skip over the character in some encodings, like at the end
+ * of the loop. But if it's a multi-byte character, it cannot
+ * have any special meaning and skipping isn't necessary for
+ * correctness. We can fall through to process it normally
+ * instead.
*/
- raw_buf_ptr++;
+ if (!IS_HIGHBIT_SET(c2))
+ raw_buf_ptr++;
+ }
}
/*
--
2.30.0
--------------97F3138F3612F1A4F9200D93--
^ permalink raw reply [nested|flat] 48+ messages in thread
* [PATCH 1/1] Fix a corner-case in COPY FROM backslash processing.
@ 2021-02-03 12:00 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 48+ messages in thread
From: Heikki Linnakangas @ 2021-02-03 12:00 UTC (permalink / raw)
If a multi-byte character is escaped with a backslash in TEXT mode input,
we didn't always treat the escape correctly. If:
- a multi-byte character is escaped with a backslash, and
- the second byte of the character is 0x5C, i.e. the ASCII code of a
backslash (\), and
- the next character is a dot (.), then
copyreadline we would incorrectly interpret the sequence as an end-of-copy
marker (\.). this can only happen in encodings that can "embed" ascii
characters as the second byte. one example of such sequence is
'\x5ca45c2e666f6f' in Big5 encoding. If you put that in a file, and
load it with COPY FROM, you'd incorrectly get an "end-of-copy marker
corrupt" error.
---
src/backend/commands/copyfromparse.c | 13 +++++++++++--
1 file changed, 11 insertions(+), 2 deletions(-)
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index 798e18e013..c20ec482db 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -1084,7 +1084,7 @@ CopyReadLineText(CopyFromState cstate)
break;
}
else if (!cstate->opts.csv_mode)
-
+ {
/*
* If we are here, it means we found a backslash followed by
* something other than a period. In non-CSV mode, anything
@@ -1095,8 +1095,17 @@ CopyReadLineText(CopyFromState cstate)
* backslashes are not special, so we want to process the
* character after the backslash just like a normal character,
* so we don't increment in those cases.
+ *
+ * In principle, we would need to use pg_encoding_mblen() to
+ * skip over the character in some encodings, like at the end
+ * of the loop. But if it's a multi-byte character, it cannot
+ * have any special meaning and skipping isn't necessary for
+ * correctness. We can fall through to process it normally
+ * instead.
*/
- raw_buf_ptr++;
+ if (!IS_HIGHBIT_SET(c2))
+ raw_buf_ptr++;
+ }
}
/*
--
2.30.0
--------------97F3138F3612F1A4F9200D93--
^ permalink raw reply [nested|flat] 48+ messages in thread
* [PATCH 1/1] Fix a corner-case in COPY FROM backslash processing.
@ 2021-02-03 12:00 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 48+ messages in thread
From: Heikki Linnakangas @ 2021-02-03 12:00 UTC (permalink / raw)
If a multi-byte character is escaped with a backslash in TEXT mode input,
we didn't always treat the escape correctly. If:
- a multi-byte character is escaped with a backslash, and
- the second byte of the character is 0x5C, i.e. the ASCII code of a
backslash (\), and
- the next character is a dot (.), then
copyreadline we would incorrectly interpret the sequence as an end-of-copy
marker (\.). this can only happen in encodings that can "embed" ascii
characters as the second byte. one example of such sequence is
'\x5ca45c2e666f6f' in Big5 encoding. If you put that in a file, and
load it with COPY FROM, you'd incorrectly get an "end-of-copy marker
corrupt" error.
---
src/backend/commands/copyfromparse.c | 13 +++++++++++--
1 file changed, 11 insertions(+), 2 deletions(-)
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index 798e18e013..c20ec482db 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -1084,7 +1084,7 @@ CopyReadLineText(CopyFromState cstate)
break;
}
else if (!cstate->opts.csv_mode)
-
+ {
/*
* If we are here, it means we found a backslash followed by
* something other than a period. In non-CSV mode, anything
@@ -1095,8 +1095,17 @@ CopyReadLineText(CopyFromState cstate)
* backslashes are not special, so we want to process the
* character after the backslash just like a normal character,
* so we don't increment in those cases.
+ *
+ * In principle, we would need to use pg_encoding_mblen() to
+ * skip over the character in some encodings, like at the end
+ * of the loop. But if it's a multi-byte character, it cannot
+ * have any special meaning and skipping isn't necessary for
+ * correctness. We can fall through to process it normally
+ * instead.
*/
- raw_buf_ptr++;
+ if (!IS_HIGHBIT_SET(c2))
+ raw_buf_ptr++;
+ }
}
/*
--
2.30.0
--------------97F3138F3612F1A4F9200D93--
^ permalink raw reply [nested|flat] 48+ messages in thread
* [PATCH 1/1] Fix a corner-case in COPY FROM backslash processing.
@ 2021-02-03 12:00 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 48+ messages in thread
From: Heikki Linnakangas @ 2021-02-03 12:00 UTC (permalink / raw)
If a multi-byte character is escaped with a backslash in TEXT mode input,
we didn't always treat the escape correctly. If:
- a multi-byte character is escaped with a backslash, and
- the second byte of the character is 0x5C, i.e. the ASCII code of a
backslash (\), and
- the next character is a dot (.), then
copyreadline we would incorrectly interpret the sequence as an end-of-copy
marker (\.). this can only happen in encodings that can "embed" ascii
characters as the second byte. one example of such sequence is
'\x5ca45c2e666f6f' in Big5 encoding. If you put that in a file, and
load it with COPY FROM, you'd incorrectly get an "end-of-copy marker
corrupt" error.
---
src/backend/commands/copyfromparse.c | 13 +++++++++++--
1 file changed, 11 insertions(+), 2 deletions(-)
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index 798e18e013..c20ec482db 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -1084,7 +1084,7 @@ CopyReadLineText(CopyFromState cstate)
break;
}
else if (!cstate->opts.csv_mode)
-
+ {
/*
* If we are here, it means we found a backslash followed by
* something other than a period. In non-CSV mode, anything
@@ -1095,8 +1095,17 @@ CopyReadLineText(CopyFromState cstate)
* backslashes are not special, so we want to process the
* character after the backslash just like a normal character,
* so we don't increment in those cases.
+ *
+ * In principle, we would need to use pg_encoding_mblen() to
+ * skip over the character in some encodings, like at the end
+ * of the loop. But if it's a multi-byte character, it cannot
+ * have any special meaning and skipping isn't necessary for
+ * correctness. We can fall through to process it normally
+ * instead.
*/
- raw_buf_ptr++;
+ if (!IS_HIGHBIT_SET(c2))
+ raw_buf_ptr++;
+ }
}
/*
--
2.30.0
--------------97F3138F3612F1A4F9200D93--
^ permalink raw reply [nested|flat] 48+ messages in thread
* [PATCH 1/1] Fix a corner-case in COPY FROM backslash processing.
@ 2021-02-03 12:00 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 48+ messages in thread
From: Heikki Linnakangas @ 2021-02-03 12:00 UTC (permalink / raw)
If a multi-byte character is escaped with a backslash in TEXT mode input,
we didn't always treat the escape correctly. If:
- a multi-byte character is escaped with a backslash, and
- the second byte of the character is 0x5C, i.e. the ASCII code of a
backslash (\), and
- the next character is a dot (.), then
copyreadline we would incorrectly interpret the sequence as an end-of-copy
marker (\.). this can only happen in encodings that can "embed" ascii
characters as the second byte. one example of such sequence is
'\x5ca45c2e666f6f' in Big5 encoding. If you put that in a file, and
load it with COPY FROM, you'd incorrectly get an "end-of-copy marker
corrupt" error.
---
src/backend/commands/copyfromparse.c | 13 +++++++++++--
1 file changed, 11 insertions(+), 2 deletions(-)
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index 798e18e013..c20ec482db 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -1084,7 +1084,7 @@ CopyReadLineText(CopyFromState cstate)
break;
}
else if (!cstate->opts.csv_mode)
-
+ {
/*
* If we are here, it means we found a backslash followed by
* something other than a period. In non-CSV mode, anything
@@ -1095,8 +1095,17 @@ CopyReadLineText(CopyFromState cstate)
* backslashes are not special, so we want to process the
* character after the backslash just like a normal character,
* so we don't increment in those cases.
+ *
+ * In principle, we would need to use pg_encoding_mblen() to
+ * skip over the character in some encodings, like at the end
+ * of the loop. But if it's a multi-byte character, it cannot
+ * have any special meaning and skipping isn't necessary for
+ * correctness. We can fall through to process it normally
+ * instead.
*/
- raw_buf_ptr++;
+ if (!IS_HIGHBIT_SET(c2))
+ raw_buf_ptr++;
+ }
}
/*
--
2.30.0
--------------97F3138F3612F1A4F9200D93--
^ permalink raw reply [nested|flat] 48+ messages in thread
* [PATCH 1/1] Fix a corner-case in COPY FROM backslash processing.
@ 2021-02-03 12:00 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 48+ messages in thread
From: Heikki Linnakangas @ 2021-02-03 12:00 UTC (permalink / raw)
If a multi-byte character is escaped with a backslash in TEXT mode input,
we didn't always treat the escape correctly. If:
- a multi-byte character is escaped with a backslash, and
- the second byte of the character is 0x5C, i.e. the ASCII code of a
backslash (\), and
- the next character is a dot (.), then
copyreadline we would incorrectly interpret the sequence as an end-of-copy
marker (\.). this can only happen in encodings that can "embed" ascii
characters as the second byte. one example of such sequence is
'\x5ca45c2e666f6f' in Big5 encoding. If you put that in a file, and
load it with COPY FROM, you'd incorrectly get an "end-of-copy marker
corrupt" error.
---
src/backend/commands/copyfromparse.c | 13 +++++++++++--
1 file changed, 11 insertions(+), 2 deletions(-)
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index 798e18e013..c20ec482db 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -1084,7 +1084,7 @@ CopyReadLineText(CopyFromState cstate)
break;
}
else if (!cstate->opts.csv_mode)
-
+ {
/*
* If we are here, it means we found a backslash followed by
* something other than a period. In non-CSV mode, anything
@@ -1095,8 +1095,17 @@ CopyReadLineText(CopyFromState cstate)
* backslashes are not special, so we want to process the
* character after the backslash just like a normal character,
* so we don't increment in those cases.
+ *
+ * In principle, we would need to use pg_encoding_mblen() to
+ * skip over the character in some encodings, like at the end
+ * of the loop. But if it's a multi-byte character, it cannot
+ * have any special meaning and skipping isn't necessary for
+ * correctness. We can fall through to process it normally
+ * instead.
*/
- raw_buf_ptr++;
+ if (!IS_HIGHBIT_SET(c2))
+ raw_buf_ptr++;
+ }
}
/*
--
2.30.0
--------------97F3138F3612F1A4F9200D93--
^ permalink raw reply [nested|flat] 48+ messages in thread
* [PATCH 1/1] Fix a corner-case in COPY FROM backslash processing.
@ 2021-02-03 12:00 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 48+ messages in thread
From: Heikki Linnakangas @ 2021-02-03 12:00 UTC (permalink / raw)
If a multi-byte character is escaped with a backslash in TEXT mode input,
we didn't always treat the escape correctly. If:
- a multi-byte character is escaped with a backslash, and
- the second byte of the character is 0x5C, i.e. the ASCII code of a
backslash (\), and
- the next character is a dot (.), then
copyreadline we would incorrectly interpret the sequence as an end-of-copy
marker (\.). this can only happen in encodings that can "embed" ascii
characters as the second byte. one example of such sequence is
'\x5ca45c2e666f6f' in Big5 encoding. If you put that in a file, and
load it with COPY FROM, you'd incorrectly get an "end-of-copy marker
corrupt" error.
---
src/backend/commands/copyfromparse.c | 13 +++++++++++--
1 file changed, 11 insertions(+), 2 deletions(-)
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index 798e18e013..c20ec482db 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -1084,7 +1084,7 @@ CopyReadLineText(CopyFromState cstate)
break;
}
else if (!cstate->opts.csv_mode)
-
+ {
/*
* If we are here, it means we found a backslash followed by
* something other than a period. In non-CSV mode, anything
@@ -1095,8 +1095,17 @@ CopyReadLineText(CopyFromState cstate)
* backslashes are not special, so we want to process the
* character after the backslash just like a normal character,
* so we don't increment in those cases.
+ *
+ * In principle, we would need to use pg_encoding_mblen() to
+ * skip over the character in some encodings, like at the end
+ * of the loop. But if it's a multi-byte character, it cannot
+ * have any special meaning and skipping isn't necessary for
+ * correctness. We can fall through to process it normally
+ * instead.
*/
- raw_buf_ptr++;
+ if (!IS_HIGHBIT_SET(c2))
+ raw_buf_ptr++;
+ }
}
/*
--
2.30.0
--------------97F3138F3612F1A4F9200D93--
^ permalink raw reply [nested|flat] 48+ messages in thread
* [PATCH 1/1] Fix a corner-case in COPY FROM backslash processing.
@ 2021-02-03 12:00 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 48+ messages in thread
From: Heikki Linnakangas @ 2021-02-03 12:00 UTC (permalink / raw)
If a multi-byte character is escaped with a backslash in TEXT mode input,
we didn't always treat the escape correctly. If:
- a multi-byte character is escaped with a backslash, and
- the second byte of the character is 0x5C, i.e. the ASCII code of a
backslash (\), and
- the next character is a dot (.), then
copyreadline we would incorrectly interpret the sequence as an end-of-copy
marker (\.). this can only happen in encodings that can "embed" ascii
characters as the second byte. one example of such sequence is
'\x5ca45c2e666f6f' in Big5 encoding. If you put that in a file, and
load it with COPY FROM, you'd incorrectly get an "end-of-copy marker
corrupt" error.
---
src/backend/commands/copyfromparse.c | 13 +++++++++++--
1 file changed, 11 insertions(+), 2 deletions(-)
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index 798e18e013..c20ec482db 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -1084,7 +1084,7 @@ CopyReadLineText(CopyFromState cstate)
break;
}
else if (!cstate->opts.csv_mode)
-
+ {
/*
* If we are here, it means we found a backslash followed by
* something other than a period. In non-CSV mode, anything
@@ -1095,8 +1095,17 @@ CopyReadLineText(CopyFromState cstate)
* backslashes are not special, so we want to process the
* character after the backslash just like a normal character,
* so we don't increment in those cases.
+ *
+ * In principle, we would need to use pg_encoding_mblen() to
+ * skip over the character in some encodings, like at the end
+ * of the loop. But if it's a multi-byte character, it cannot
+ * have any special meaning and skipping isn't necessary for
+ * correctness. We can fall through to process it normally
+ * instead.
*/
- raw_buf_ptr++;
+ if (!IS_HIGHBIT_SET(c2))
+ raw_buf_ptr++;
+ }
}
/*
--
2.30.0
--------------97F3138F3612F1A4F9200D93--
^ permalink raw reply [nested|flat] 48+ messages in thread
* [PATCH 1/1] Fix a corner-case in COPY FROM backslash processing.
@ 2021-02-03 12:00 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 48+ messages in thread
From: Heikki Linnakangas @ 2021-02-03 12:00 UTC (permalink / raw)
If a multi-byte character is escaped with a backslash in TEXT mode input,
we didn't always treat the escape correctly. If:
- a multi-byte character is escaped with a backslash, and
- the second byte of the character is 0x5C, i.e. the ASCII code of a
backslash (\), and
- the next character is a dot (.), then
copyreadline we would incorrectly interpret the sequence as an end-of-copy
marker (\.). this can only happen in encodings that can "embed" ascii
characters as the second byte. one example of such sequence is
'\x5ca45c2e666f6f' in Big5 encoding. If you put that in a file, and
load it with COPY FROM, you'd incorrectly get an "end-of-copy marker
corrupt" error.
---
src/backend/commands/copyfromparse.c | 13 +++++++++++--
1 file changed, 11 insertions(+), 2 deletions(-)
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index 798e18e013..c20ec482db 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -1084,7 +1084,7 @@ CopyReadLineText(CopyFromState cstate)
break;
}
else if (!cstate->opts.csv_mode)
-
+ {
/*
* If we are here, it means we found a backslash followed by
* something other than a period. In non-CSV mode, anything
@@ -1095,8 +1095,17 @@ CopyReadLineText(CopyFromState cstate)
* backslashes are not special, so we want to process the
* character after the backslash just like a normal character,
* so we don't increment in those cases.
+ *
+ * In principle, we would need to use pg_encoding_mblen() to
+ * skip over the character in some encodings, like at the end
+ * of the loop. But if it's a multi-byte character, it cannot
+ * have any special meaning and skipping isn't necessary for
+ * correctness. We can fall through to process it normally
+ * instead.
*/
- raw_buf_ptr++;
+ if (!IS_HIGHBIT_SET(c2))
+ raw_buf_ptr++;
+ }
}
/*
--
2.30.0
--------------97F3138F3612F1A4F9200D93--
^ permalink raw reply [nested|flat] 48+ messages in thread
* [PATCH 1/1] Fix a corner-case in COPY FROM backslash processing.
@ 2021-02-03 12:00 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 48+ messages in thread
From: Heikki Linnakangas @ 2021-02-03 12:00 UTC (permalink / raw)
If a multi-byte character is escaped with a backslash in TEXT mode input,
we didn't always treat the escape correctly. If:
- a multi-byte character is escaped with a backslash, and
- the second byte of the character is 0x5C, i.e. the ASCII code of a
backslash (\), and
- the next character is a dot (.), then
copyreadline we would incorrectly interpret the sequence as an end-of-copy
marker (\.). this can only happen in encodings that can "embed" ascii
characters as the second byte. one example of such sequence is
'\x5ca45c2e666f6f' in Big5 encoding. If you put that in a file, and
load it with COPY FROM, you'd incorrectly get an "end-of-copy marker
corrupt" error.
---
src/backend/commands/copyfromparse.c | 13 +++++++++++--
1 file changed, 11 insertions(+), 2 deletions(-)
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index 798e18e013..c20ec482db 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -1084,7 +1084,7 @@ CopyReadLineText(CopyFromState cstate)
break;
}
else if (!cstate->opts.csv_mode)
-
+ {
/*
* If we are here, it means we found a backslash followed by
* something other than a period. In non-CSV mode, anything
@@ -1095,8 +1095,17 @@ CopyReadLineText(CopyFromState cstate)
* backslashes are not special, so we want to process the
* character after the backslash just like a normal character,
* so we don't increment in those cases.
+ *
+ * In principle, we would need to use pg_encoding_mblen() to
+ * skip over the character in some encodings, like at the end
+ * of the loop. But if it's a multi-byte character, it cannot
+ * have any special meaning and skipping isn't necessary for
+ * correctness. We can fall through to process it normally
+ * instead.
*/
- raw_buf_ptr++;
+ if (!IS_HIGHBIT_SET(c2))
+ raw_buf_ptr++;
+ }
}
/*
--
2.30.0
--------------97F3138F3612F1A4F9200D93--
^ permalink raw reply [nested|flat] 48+ messages in thread
* [PATCH 1/1] Fix a corner-case in COPY FROM backslash processing.
@ 2021-02-03 12:00 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 48+ messages in thread
From: Heikki Linnakangas @ 2021-02-03 12:00 UTC (permalink / raw)
If a multi-byte character is escaped with a backslash in TEXT mode input,
we didn't always treat the escape correctly. If:
- a multi-byte character is escaped with a backslash, and
- the second byte of the character is 0x5C, i.e. the ASCII code of a
backslash (\), and
- the next character is a dot (.), then
copyreadline we would incorrectly interpret the sequence as an end-of-copy
marker (\.). this can only happen in encodings that can "embed" ascii
characters as the second byte. one example of such sequence is
'\x5ca45c2e666f6f' in Big5 encoding. If you put that in a file, and
load it with COPY FROM, you'd incorrectly get an "end-of-copy marker
corrupt" error.
---
src/backend/commands/copyfromparse.c | 13 +++++++++++--
1 file changed, 11 insertions(+), 2 deletions(-)
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index 798e18e013..c20ec482db 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -1084,7 +1084,7 @@ CopyReadLineText(CopyFromState cstate)
break;
}
else if (!cstate->opts.csv_mode)
-
+ {
/*
* If we are here, it means we found a backslash followed by
* something other than a period. In non-CSV mode, anything
@@ -1095,8 +1095,17 @@ CopyReadLineText(CopyFromState cstate)
* backslashes are not special, so we want to process the
* character after the backslash just like a normal character,
* so we don't increment in those cases.
+ *
+ * In principle, we would need to use pg_encoding_mblen() to
+ * skip over the character in some encodings, like at the end
+ * of the loop. But if it's a multi-byte character, it cannot
+ * have any special meaning and skipping isn't necessary for
+ * correctness. We can fall through to process it normally
+ * instead.
*/
- raw_buf_ptr++;
+ if (!IS_HIGHBIT_SET(c2))
+ raw_buf_ptr++;
+ }
}
/*
--
2.30.0
--------------97F3138F3612F1A4F9200D93--
^ permalink raw reply [nested|flat] 48+ messages in thread
* [PATCH 1/1] Fix a corner-case in COPY FROM backslash processing.
@ 2021-02-03 12:00 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 48+ messages in thread
From: Heikki Linnakangas @ 2021-02-03 12:00 UTC (permalink / raw)
If a multi-byte character is escaped with a backslash in TEXT mode input,
we didn't always treat the escape correctly. If:
- a multi-byte character is escaped with a backslash, and
- the second byte of the character is 0x5C, i.e. the ASCII code of a
backslash (\), and
- the next character is a dot (.), then
copyreadline we would incorrectly interpret the sequence as an end-of-copy
marker (\.). this can only happen in encodings that can "embed" ascii
characters as the second byte. one example of such sequence is
'\x5ca45c2e666f6f' in Big5 encoding. If you put that in a file, and
load it with COPY FROM, you'd incorrectly get an "end-of-copy marker
corrupt" error.
---
src/backend/commands/copyfromparse.c | 13 +++++++++++--
1 file changed, 11 insertions(+), 2 deletions(-)
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index 798e18e013..c20ec482db 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -1084,7 +1084,7 @@ CopyReadLineText(CopyFromState cstate)
break;
}
else if (!cstate->opts.csv_mode)
-
+ {
/*
* If we are here, it means we found a backslash followed by
* something other than a period. In non-CSV mode, anything
@@ -1095,8 +1095,17 @@ CopyReadLineText(CopyFromState cstate)
* backslashes are not special, so we want to process the
* character after the backslash just like a normal character,
* so we don't increment in those cases.
+ *
+ * In principle, we would need to use pg_encoding_mblen() to
+ * skip over the character in some encodings, like at the end
+ * of the loop. But if it's a multi-byte character, it cannot
+ * have any special meaning and skipping isn't necessary for
+ * correctness. We can fall through to process it normally
+ * instead.
*/
- raw_buf_ptr++;
+ if (!IS_HIGHBIT_SET(c2))
+ raw_buf_ptr++;
+ }
}
/*
--
2.30.0
--------------97F3138F3612F1A4F9200D93--
^ permalink raw reply [nested|flat] 48+ messages in thread
* [PATCH 1/1] Fix a corner-case in COPY FROM backslash processing.
@ 2021-02-03 12:00 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 48+ messages in thread
From: Heikki Linnakangas @ 2021-02-03 12:00 UTC (permalink / raw)
If a multi-byte character is escaped with a backslash in TEXT mode input,
we didn't always treat the escape correctly. If:
- a multi-byte character is escaped with a backslash, and
- the second byte of the character is 0x5C, i.e. the ASCII code of a
backslash (\), and
- the next character is a dot (.), then
copyreadline we would incorrectly interpret the sequence as an end-of-copy
marker (\.). this can only happen in encodings that can "embed" ascii
characters as the second byte. one example of such sequence is
'\x5ca45c2e666f6f' in Big5 encoding. If you put that in a file, and
load it with COPY FROM, you'd incorrectly get an "end-of-copy marker
corrupt" error.
---
src/backend/commands/copyfromparse.c | 13 +++++++++++--
1 file changed, 11 insertions(+), 2 deletions(-)
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index 798e18e013..c20ec482db 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -1084,7 +1084,7 @@ CopyReadLineText(CopyFromState cstate)
break;
}
else if (!cstate->opts.csv_mode)
-
+ {
/*
* If we are here, it means we found a backslash followed by
* something other than a period. In non-CSV mode, anything
@@ -1095,8 +1095,17 @@ CopyReadLineText(CopyFromState cstate)
* backslashes are not special, so we want to process the
* character after the backslash just like a normal character,
* so we don't increment in those cases.
+ *
+ * In principle, we would need to use pg_encoding_mblen() to
+ * skip over the character in some encodings, like at the end
+ * of the loop. But if it's a multi-byte character, it cannot
+ * have any special meaning and skipping isn't necessary for
+ * correctness. We can fall through to process it normally
+ * instead.
*/
- raw_buf_ptr++;
+ if (!IS_HIGHBIT_SET(c2))
+ raw_buf_ptr++;
+ }
}
/*
--
2.30.0
--------------97F3138F3612F1A4F9200D93--
^ permalink raw reply [nested|flat] 48+ messages in thread
* [PATCH 1/1] Fix a corner-case in COPY FROM backslash processing.
@ 2021-02-03 12:00 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 48+ messages in thread
From: Heikki Linnakangas @ 2021-02-03 12:00 UTC (permalink / raw)
If a multi-byte character is escaped with a backslash in TEXT mode input,
we didn't always treat the escape correctly. If:
- a multi-byte character is escaped with a backslash, and
- the second byte of the character is 0x5C, i.e. the ASCII code of a
backslash (\), and
- the next character is a dot (.), then
copyreadline we would incorrectly interpret the sequence as an end-of-copy
marker (\.). this can only happen in encodings that can "embed" ascii
characters as the second byte. one example of such sequence is
'\x5ca45c2e666f6f' in Big5 encoding. If you put that in a file, and
load it with COPY FROM, you'd incorrectly get an "end-of-copy marker
corrupt" error.
---
src/backend/commands/copyfromparse.c | 13 +++++++++++--
1 file changed, 11 insertions(+), 2 deletions(-)
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index 798e18e013..c20ec482db 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -1084,7 +1084,7 @@ CopyReadLineText(CopyFromState cstate)
break;
}
else if (!cstate->opts.csv_mode)
-
+ {
/*
* If we are here, it means we found a backslash followed by
* something other than a period. In non-CSV mode, anything
@@ -1095,8 +1095,17 @@ CopyReadLineText(CopyFromState cstate)
* backslashes are not special, so we want to process the
* character after the backslash just like a normal character,
* so we don't increment in those cases.
+ *
+ * In principle, we would need to use pg_encoding_mblen() to
+ * skip over the character in some encodings, like at the end
+ * of the loop. But if it's a multi-byte character, it cannot
+ * have any special meaning and skipping isn't necessary for
+ * correctness. We can fall through to process it normally
+ * instead.
*/
- raw_buf_ptr++;
+ if (!IS_HIGHBIT_SET(c2))
+ raw_buf_ptr++;
+ }
}
/*
--
2.30.0
--------------97F3138F3612F1A4F9200D93--
^ permalink raw reply [nested|flat] 48+ messages in thread
* [PATCH 1/1] Fix a corner-case in COPY FROM backslash processing.
@ 2021-02-03 12:00 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 48+ messages in thread
From: Heikki Linnakangas @ 2021-02-03 12:00 UTC (permalink / raw)
If a multi-byte character is escaped with a backslash in TEXT mode input,
we didn't always treat the escape correctly. If:
- a multi-byte character is escaped with a backslash, and
- the second byte of the character is 0x5C, i.e. the ASCII code of a
backslash (\), and
- the next character is a dot (.), then
copyreadline we would incorrectly interpret the sequence as an end-of-copy
marker (\.). this can only happen in encodings that can "embed" ascii
characters as the second byte. one example of such sequence is
'\x5ca45c2e666f6f' in Big5 encoding. If you put that in a file, and
load it with COPY FROM, you'd incorrectly get an "end-of-copy marker
corrupt" error.
---
src/backend/commands/copyfromparse.c | 13 +++++++++++--
1 file changed, 11 insertions(+), 2 deletions(-)
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index 798e18e013..c20ec482db 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -1084,7 +1084,7 @@ CopyReadLineText(CopyFromState cstate)
break;
}
else if (!cstate->opts.csv_mode)
-
+ {
/*
* If we are here, it means we found a backslash followed by
* something other than a period. In non-CSV mode, anything
@@ -1095,8 +1095,17 @@ CopyReadLineText(CopyFromState cstate)
* backslashes are not special, so we want to process the
* character after the backslash just like a normal character,
* so we don't increment in those cases.
+ *
+ * In principle, we would need to use pg_encoding_mblen() to
+ * skip over the character in some encodings, like at the end
+ * of the loop. But if it's a multi-byte character, it cannot
+ * have any special meaning and skipping isn't necessary for
+ * correctness. We can fall through to process it normally
+ * instead.
*/
- raw_buf_ptr++;
+ if (!IS_HIGHBIT_SET(c2))
+ raw_buf_ptr++;
+ }
}
/*
--
2.30.0
--------------97F3138F3612F1A4F9200D93--
^ permalink raw reply [nested|flat] 48+ messages in thread
* [PATCH 1/1] Fix a corner-case in COPY FROM backslash processing.
@ 2021-02-03 12:00 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 48+ messages in thread
From: Heikki Linnakangas @ 2021-02-03 12:00 UTC (permalink / raw)
If a multi-byte character is escaped with a backslash in TEXT mode input,
we didn't always treat the escape correctly. If:
- a multi-byte character is escaped with a backslash, and
- the second byte of the character is 0x5C, i.e. the ASCII code of a
backslash (\), and
- the next character is a dot (.), then
copyreadline we would incorrectly interpret the sequence as an end-of-copy
marker (\.). this can only happen in encodings that can "embed" ascii
characters as the second byte. one example of such sequence is
'\x5ca45c2e666f6f' in Big5 encoding. If you put that in a file, and
load it with COPY FROM, you'd incorrectly get an "end-of-copy marker
corrupt" error.
---
src/backend/commands/copyfromparse.c | 13 +++++++++++--
1 file changed, 11 insertions(+), 2 deletions(-)
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index 798e18e013..c20ec482db 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -1084,7 +1084,7 @@ CopyReadLineText(CopyFromState cstate)
break;
}
else if (!cstate->opts.csv_mode)
-
+ {
/*
* If we are here, it means we found a backslash followed by
* something other than a period. In non-CSV mode, anything
@@ -1095,8 +1095,17 @@ CopyReadLineText(CopyFromState cstate)
* backslashes are not special, so we want to process the
* character after the backslash just like a normal character,
* so we don't increment in those cases.
+ *
+ * In principle, we would need to use pg_encoding_mblen() to
+ * skip over the character in some encodings, like at the end
+ * of the loop. But if it's a multi-byte character, it cannot
+ * have any special meaning and skipping isn't necessary for
+ * correctness. We can fall through to process it normally
+ * instead.
*/
- raw_buf_ptr++;
+ if (!IS_HIGHBIT_SET(c2))
+ raw_buf_ptr++;
+ }
}
/*
--
2.30.0
--------------97F3138F3612F1A4F9200D93--
^ permalink raw reply [nested|flat] 48+ messages in thread
* [PATCH 1/1] Fix a corner-case in COPY FROM backslash processing.
@ 2021-02-03 12:00 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 48+ messages in thread
From: Heikki Linnakangas @ 2021-02-03 12:00 UTC (permalink / raw)
If a multi-byte character is escaped with a backslash in TEXT mode input,
we didn't always treat the escape correctly. If:
- a multi-byte character is escaped with a backslash, and
- the second byte of the character is 0x5C, i.e. the ASCII code of a
backslash (\), and
- the next character is a dot (.), then
copyreadline we would incorrectly interpret the sequence as an end-of-copy
marker (\.). this can only happen in encodings that can "embed" ascii
characters as the second byte. one example of such sequence is
'\x5ca45c2e666f6f' in Big5 encoding. If you put that in a file, and
load it with COPY FROM, you'd incorrectly get an "end-of-copy marker
corrupt" error.
---
src/backend/commands/copyfromparse.c | 13 +++++++++++--
1 file changed, 11 insertions(+), 2 deletions(-)
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index 798e18e013..c20ec482db 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -1084,7 +1084,7 @@ CopyReadLineText(CopyFromState cstate)
break;
}
else if (!cstate->opts.csv_mode)
-
+ {
/*
* If we are here, it means we found a backslash followed by
* something other than a period. In non-CSV mode, anything
@@ -1095,8 +1095,17 @@ CopyReadLineText(CopyFromState cstate)
* backslashes are not special, so we want to process the
* character after the backslash just like a normal character,
* so we don't increment in those cases.
+ *
+ * In principle, we would need to use pg_encoding_mblen() to
+ * skip over the character in some encodings, like at the end
+ * of the loop. But if it's a multi-byte character, it cannot
+ * have any special meaning and skipping isn't necessary for
+ * correctness. We can fall through to process it normally
+ * instead.
*/
- raw_buf_ptr++;
+ if (!IS_HIGHBIT_SET(c2))
+ raw_buf_ptr++;
+ }
}
/*
--
2.30.0
--------------97F3138F3612F1A4F9200D93--
^ permalink raw reply [nested|flat] 48+ messages in thread
* [PATCH 1/1] Fix a corner-case in COPY FROM backslash processing.
@ 2021-02-03 12:00 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 48+ messages in thread
From: Heikki Linnakangas @ 2021-02-03 12:00 UTC (permalink / raw)
If a multi-byte character is escaped with a backslash in TEXT mode input,
we didn't always treat the escape correctly. If:
- a multi-byte character is escaped with a backslash, and
- the second byte of the character is 0x5C, i.e. the ASCII code of a
backslash (\), and
- the next character is a dot (.), then
copyreadline we would incorrectly interpret the sequence as an end-of-copy
marker (\.). this can only happen in encodings that can "embed" ascii
characters as the second byte. one example of such sequence is
'\x5ca45c2e666f6f' in Big5 encoding. If you put that in a file, and
load it with COPY FROM, you'd incorrectly get an "end-of-copy marker
corrupt" error.
---
src/backend/commands/copyfromparse.c | 13 +++++++++++--
1 file changed, 11 insertions(+), 2 deletions(-)
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index 798e18e013..c20ec482db 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -1084,7 +1084,7 @@ CopyReadLineText(CopyFromState cstate)
break;
}
else if (!cstate->opts.csv_mode)
-
+ {
/*
* If we are here, it means we found a backslash followed by
* something other than a period. In non-CSV mode, anything
@@ -1095,8 +1095,17 @@ CopyReadLineText(CopyFromState cstate)
* backslashes are not special, so we want to process the
* character after the backslash just like a normal character,
* so we don't increment in those cases.
+ *
+ * In principle, we would need to use pg_encoding_mblen() to
+ * skip over the character in some encodings, like at the end
+ * of the loop. But if it's a multi-byte character, it cannot
+ * have any special meaning and skipping isn't necessary for
+ * correctness. We can fall through to process it normally
+ * instead.
*/
- raw_buf_ptr++;
+ if (!IS_HIGHBIT_SET(c2))
+ raw_buf_ptr++;
+ }
}
/*
--
2.30.0
--------------97F3138F3612F1A4F9200D93--
^ permalink raw reply [nested|flat] 48+ messages in thread
* [PATCH 1/1] Fix a corner-case in COPY FROM backslash processing.
@ 2021-02-03 12:00 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 48+ messages in thread
From: Heikki Linnakangas @ 2021-02-03 12:00 UTC (permalink / raw)
If a multi-byte character is escaped with a backslash in TEXT mode input,
we didn't always treat the escape correctly. If:
- a multi-byte character is escaped with a backslash, and
- the second byte of the character is 0x5C, i.e. the ASCII code of a
backslash (\), and
- the next character is a dot (.), then
copyreadline we would incorrectly interpret the sequence as an end-of-copy
marker (\.). this can only happen in encodings that can "embed" ascii
characters as the second byte. one example of such sequence is
'\x5ca45c2e666f6f' in Big5 encoding. If you put that in a file, and
load it with COPY FROM, you'd incorrectly get an "end-of-copy marker
corrupt" error.
---
src/backend/commands/copyfromparse.c | 13 +++++++++++--
1 file changed, 11 insertions(+), 2 deletions(-)
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index 798e18e013..c20ec482db 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -1084,7 +1084,7 @@ CopyReadLineText(CopyFromState cstate)
break;
}
else if (!cstate->opts.csv_mode)
-
+ {
/*
* If we are here, it means we found a backslash followed by
* something other than a period. In non-CSV mode, anything
@@ -1095,8 +1095,17 @@ CopyReadLineText(CopyFromState cstate)
* backslashes are not special, so we want to process the
* character after the backslash just like a normal character,
* so we don't increment in those cases.
+ *
+ * In principle, we would need to use pg_encoding_mblen() to
+ * skip over the character in some encodings, like at the end
+ * of the loop. But if it's a multi-byte character, it cannot
+ * have any special meaning and skipping isn't necessary for
+ * correctness. We can fall through to process it normally
+ * instead.
*/
- raw_buf_ptr++;
+ if (!IS_HIGHBIT_SET(c2))
+ raw_buf_ptr++;
+ }
}
/*
--
2.30.0
--------------97F3138F3612F1A4F9200D93--
^ permalink raw reply [nested|flat] 48+ messages in thread
* [PATCH 1/1] Fix a corner-case in COPY FROM backslash processing.
@ 2021-02-03 12:00 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 48+ messages in thread
From: Heikki Linnakangas @ 2021-02-03 12:00 UTC (permalink / raw)
If a multi-byte character is escaped with a backslash in TEXT mode input,
we didn't always treat the escape correctly. If:
- a multi-byte character is escaped with a backslash, and
- the second byte of the character is 0x5C, i.e. the ASCII code of a
backslash (\), and
- the next character is a dot (.), then
copyreadline we would incorrectly interpret the sequence as an end-of-copy
marker (\.). this can only happen in encodings that can "embed" ascii
characters as the second byte. one example of such sequence is
'\x5ca45c2e666f6f' in Big5 encoding. If you put that in a file, and
load it with COPY FROM, you'd incorrectly get an "end-of-copy marker
corrupt" error.
---
src/backend/commands/copyfromparse.c | 13 +++++++++++--
1 file changed, 11 insertions(+), 2 deletions(-)
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index 798e18e013..c20ec482db 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -1084,7 +1084,7 @@ CopyReadLineText(CopyFromState cstate)
break;
}
else if (!cstate->opts.csv_mode)
-
+ {
/*
* If we are here, it means we found a backslash followed by
* something other than a period. In non-CSV mode, anything
@@ -1095,8 +1095,17 @@ CopyReadLineText(CopyFromState cstate)
* backslashes are not special, so we want to process the
* character after the backslash just like a normal character,
* so we don't increment in those cases.
+ *
+ * In principle, we would need to use pg_encoding_mblen() to
+ * skip over the character in some encodings, like at the end
+ * of the loop. But if it's a multi-byte character, it cannot
+ * have any special meaning and skipping isn't necessary for
+ * correctness. We can fall through to process it normally
+ * instead.
*/
- raw_buf_ptr++;
+ if (!IS_HIGHBIT_SET(c2))
+ raw_buf_ptr++;
+ }
}
/*
--
2.30.0
--------------97F3138F3612F1A4F9200D93--
^ permalink raw reply [nested|flat] 48+ messages in thread
* [PATCH v2 1/1] Fix a corner-case in COPY FROM backslash processing.
@ 2021-02-04 19:36 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 48+ messages in thread
From: Heikki Linnakangas @ 2021-02-04 19:36 UTC (permalink / raw)
If a multi-byte character is escaped with a backslash in TEXT mode input,
we didn't always treat the escape correctly. If:
- a multi-byte character is escaped with a backslash, and
- the second byte of the character is 0x5C, i.e. the ASCII code of a
backslash (\), and
- the next character is a dot (.), then
CopyReadLineText function would incorrectly interpret the sequence as an
end-of-copy marker (\.). This can only happen in encodings that can
"embed" ascii characters as the second byte. One example of such sequence
is '\x5ca45c2e666f6f' in Big5 encoding. If you put that in a file, and
load it with COPY FROM, you'd incorrectly get an "end-of-copy marker
corrupt" error.
Backpatch to all supported versions.
Reviewed-by: John Naylor, Kyotaro Horiguchi
Discussion: https://www.postgresql.org/message-id/a897f84f-8dca-8798-3139-07da5bb38728%40iki.fi
---
src/backend/commands/copyfromparse.c | 10 +++++++++-
1 file changed, 9 insertions(+), 1 deletion(-)
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index b843d315b1..315b16fd7a 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -1084,7 +1084,7 @@ CopyReadLineText(CopyFromState cstate)
break;
}
else if (!cstate->opts.csv_mode)
-
+ {
/*
* If we are here, it means we found a backslash followed by
* something other than a period. In non-CSV mode, anything
@@ -1095,8 +1095,16 @@ CopyReadLineText(CopyFromState cstate)
* backslashes are not special, so we want to process the
* character after the backslash just like a normal character,
* so we don't increment in those cases.
+ *
+ * Set 'c' to skip whole character correctly in multi-byte
+ * encodings. If we don't have the whole character in the
+ * buffer yet, we might loop back to process it, after all,
+ * but that's OK because multi-byte characters cannot have any
+ * special meaning.
*/
raw_buf_ptr++;
+ c = c2;
+ }
}
/*
--
2.30.0
--------------CFC80B40DFDD66DF067F288D--
^ permalink raw reply [nested|flat] 48+ messages in thread
* [PATCH v2 1/1] Fix a corner-case in COPY FROM backslash processing.
@ 2021-02-04 19:36 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 48+ messages in thread
From: Heikki Linnakangas @ 2021-02-04 19:36 UTC (permalink / raw)
If a multi-byte character is escaped with a backslash in TEXT mode input,
we didn't always treat the escape correctly. If:
- a multi-byte character is escaped with a backslash, and
- the second byte of the character is 0x5C, i.e. the ASCII code of a
backslash (\), and
- the next character is a dot (.), then
CopyReadLineText function would incorrectly interpret the sequence as an
end-of-copy marker (\.). This can only happen in encodings that can
"embed" ascii characters as the second byte. One example of such sequence
is '\x5ca45c2e666f6f' in Big5 encoding. If you put that in a file, and
load it with COPY FROM, you'd incorrectly get an "end-of-copy marker
corrupt" error.
Backpatch to all supported versions.
Reviewed-by: John Naylor, Kyotaro Horiguchi
Discussion: https://www.postgresql.org/message-id/a897f84f-8dca-8798-3139-07da5bb38728%40iki.fi
---
src/backend/commands/copyfromparse.c | 10 +++++++++-
1 file changed, 9 insertions(+), 1 deletion(-)
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index b843d315b1..315b16fd7a 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -1084,7 +1084,7 @@ CopyReadLineText(CopyFromState cstate)
break;
}
else if (!cstate->opts.csv_mode)
-
+ {
/*
* If we are here, it means we found a backslash followed by
* something other than a period. In non-CSV mode, anything
@@ -1095,8 +1095,16 @@ CopyReadLineText(CopyFromState cstate)
* backslashes are not special, so we want to process the
* character after the backslash just like a normal character,
* so we don't increment in those cases.
+ *
+ * Set 'c' to skip whole character correctly in multi-byte
+ * encodings. If we don't have the whole character in the
+ * buffer yet, we might loop back to process it, after all,
+ * but that's OK because multi-byte characters cannot have any
+ * special meaning.
*/
raw_buf_ptr++;
+ c = c2;
+ }
}
/*
--
2.30.0
--------------CFC80B40DFDD66DF067F288D--
^ permalink raw reply [nested|flat] 48+ messages in thread
* [PATCH v2 1/1] Fix a corner-case in COPY FROM backslash processing.
@ 2021-02-04 19:36 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 48+ messages in thread
From: Heikki Linnakangas @ 2021-02-04 19:36 UTC (permalink / raw)
If a multi-byte character is escaped with a backslash in TEXT mode input,
we didn't always treat the escape correctly. If:
- a multi-byte character is escaped with a backslash, and
- the second byte of the character is 0x5C, i.e. the ASCII code of a
backslash (\), and
- the next character is a dot (.), then
CopyReadLineText function would incorrectly interpret the sequence as an
end-of-copy marker (\.). This can only happen in encodings that can
"embed" ascii characters as the second byte. One example of such sequence
is '\x5ca45c2e666f6f' in Big5 encoding. If you put that in a file, and
load it with COPY FROM, you'd incorrectly get an "end-of-copy marker
corrupt" error.
Backpatch to all supported versions.
Reviewed-by: John Naylor, Kyotaro Horiguchi
Discussion: https://www.postgresql.org/message-id/a897f84f-8dca-8798-3139-07da5bb38728%40iki.fi
---
src/backend/commands/copyfromparse.c | 10 +++++++++-
1 file changed, 9 insertions(+), 1 deletion(-)
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index b843d315b1..315b16fd7a 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -1084,7 +1084,7 @@ CopyReadLineText(CopyFromState cstate)
break;
}
else if (!cstate->opts.csv_mode)
-
+ {
/*
* If we are here, it means we found a backslash followed by
* something other than a period. In non-CSV mode, anything
@@ -1095,8 +1095,16 @@ CopyReadLineText(CopyFromState cstate)
* backslashes are not special, so we want to process the
* character after the backslash just like a normal character,
* so we don't increment in those cases.
+ *
+ * Set 'c' to skip whole character correctly in multi-byte
+ * encodings. If we don't have the whole character in the
+ * buffer yet, we might loop back to process it, after all,
+ * but that's OK because multi-byte characters cannot have any
+ * special meaning.
*/
raw_buf_ptr++;
+ c = c2;
+ }
}
/*
--
2.30.0
--------------CFC80B40DFDD66DF067F288D--
^ permalink raw reply [nested|flat] 48+ messages in thread
* [PATCH v2 1/1] Fix a corner-case in COPY FROM backslash processing.
@ 2021-02-04 19:36 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 48+ messages in thread
From: Heikki Linnakangas @ 2021-02-04 19:36 UTC (permalink / raw)
If a multi-byte character is escaped with a backslash in TEXT mode input,
we didn't always treat the escape correctly. If:
- a multi-byte character is escaped with a backslash, and
- the second byte of the character is 0x5C, i.e. the ASCII code of a
backslash (\), and
- the next character is a dot (.), then
CopyReadLineText function would incorrectly interpret the sequence as an
end-of-copy marker (\.). This can only happen in encodings that can
"embed" ascii characters as the second byte. One example of such sequence
is '\x5ca45c2e666f6f' in Big5 encoding. If you put that in a file, and
load it with COPY FROM, you'd incorrectly get an "end-of-copy marker
corrupt" error.
Backpatch to all supported versions.
Reviewed-by: John Naylor, Kyotaro Horiguchi
Discussion: https://www.postgresql.org/message-id/a897f84f-8dca-8798-3139-07da5bb38728%40iki.fi
---
src/backend/commands/copyfromparse.c | 10 +++++++++-
1 file changed, 9 insertions(+), 1 deletion(-)
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index b843d315b1..315b16fd7a 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -1084,7 +1084,7 @@ CopyReadLineText(CopyFromState cstate)
break;
}
else if (!cstate->opts.csv_mode)
-
+ {
/*
* If we are here, it means we found a backslash followed by
* something other than a period. In non-CSV mode, anything
@@ -1095,8 +1095,16 @@ CopyReadLineText(CopyFromState cstate)
* backslashes are not special, so we want to process the
* character after the backslash just like a normal character,
* so we don't increment in those cases.
+ *
+ * Set 'c' to skip whole character correctly in multi-byte
+ * encodings. If we don't have the whole character in the
+ * buffer yet, we might loop back to process it, after all,
+ * but that's OK because multi-byte characters cannot have any
+ * special meaning.
*/
raw_buf_ptr++;
+ c = c2;
+ }
}
/*
--
2.30.0
--------------CFC80B40DFDD66DF067F288D--
^ permalink raw reply [nested|flat] 48+ messages in thread
* [PATCH v2 1/1] Fix a corner-case in COPY FROM backslash processing.
@ 2021-02-04 19:36 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 48+ messages in thread
From: Heikki Linnakangas @ 2021-02-04 19:36 UTC (permalink / raw)
If a multi-byte character is escaped with a backslash in TEXT mode input,
we didn't always treat the escape correctly. If:
- a multi-byte character is escaped with a backslash, and
- the second byte of the character is 0x5C, i.e. the ASCII code of a
backslash (\), and
- the next character is a dot (.), then
CopyReadLineText function would incorrectly interpret the sequence as an
end-of-copy marker (\.). This can only happen in encodings that can
"embed" ascii characters as the second byte. One example of such sequence
is '\x5ca45c2e666f6f' in Big5 encoding. If you put that in a file, and
load it with COPY FROM, you'd incorrectly get an "end-of-copy marker
corrupt" error.
Backpatch to all supported versions.
Reviewed-by: John Naylor, Kyotaro Horiguchi
Discussion: https://www.postgresql.org/message-id/a897f84f-8dca-8798-3139-07da5bb38728%40iki.fi
---
src/backend/commands/copyfromparse.c | 10 +++++++++-
1 file changed, 9 insertions(+), 1 deletion(-)
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index b843d315b1..315b16fd7a 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -1084,7 +1084,7 @@ CopyReadLineText(CopyFromState cstate)
break;
}
else if (!cstate->opts.csv_mode)
-
+ {
/*
* If we are here, it means we found a backslash followed by
* something other than a period. In non-CSV mode, anything
@@ -1095,8 +1095,16 @@ CopyReadLineText(CopyFromState cstate)
* backslashes are not special, so we want to process the
* character after the backslash just like a normal character,
* so we don't increment in those cases.
+ *
+ * Set 'c' to skip whole character correctly in multi-byte
+ * encodings. If we don't have the whole character in the
+ * buffer yet, we might loop back to process it, after all,
+ * but that's OK because multi-byte characters cannot have any
+ * special meaning.
*/
raw_buf_ptr++;
+ c = c2;
+ }
}
/*
--
2.30.0
--------------CFC80B40DFDD66DF067F288D--
^ permalink raw reply [nested|flat] 48+ messages in thread
* [PATCH v2 1/1] Fix a corner-case in COPY FROM backslash processing.
@ 2021-02-04 19:36 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 48+ messages in thread
From: Heikki Linnakangas @ 2021-02-04 19:36 UTC (permalink / raw)
If a multi-byte character is escaped with a backslash in TEXT mode input,
we didn't always treat the escape correctly. If:
- a multi-byte character is escaped with a backslash, and
- the second byte of the character is 0x5C, i.e. the ASCII code of a
backslash (\), and
- the next character is a dot (.), then
CopyReadLineText function would incorrectly interpret the sequence as an
end-of-copy marker (\.). This can only happen in encodings that can
"embed" ascii characters as the second byte. One example of such sequence
is '\x5ca45c2e666f6f' in Big5 encoding. If you put that in a file, and
load it with COPY FROM, you'd incorrectly get an "end-of-copy marker
corrupt" error.
Backpatch to all supported versions.
Reviewed-by: John Naylor, Kyotaro Horiguchi
Discussion: https://www.postgresql.org/message-id/a897f84f-8dca-8798-3139-07da5bb38728%40iki.fi
---
src/backend/commands/copyfromparse.c | 10 +++++++++-
1 file changed, 9 insertions(+), 1 deletion(-)
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index b843d315b1..315b16fd7a 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -1084,7 +1084,7 @@ CopyReadLineText(CopyFromState cstate)
break;
}
else if (!cstate->opts.csv_mode)
-
+ {
/*
* If we are here, it means we found a backslash followed by
* something other than a period. In non-CSV mode, anything
@@ -1095,8 +1095,16 @@ CopyReadLineText(CopyFromState cstate)
* backslashes are not special, so we want to process the
* character after the backslash just like a normal character,
* so we don't increment in those cases.
+ *
+ * Set 'c' to skip whole character correctly in multi-byte
+ * encodings. If we don't have the whole character in the
+ * buffer yet, we might loop back to process it, after all,
+ * but that's OK because multi-byte characters cannot have any
+ * special meaning.
*/
raw_buf_ptr++;
+ c = c2;
+ }
}
/*
--
2.30.0
--------------CFC80B40DFDD66DF067F288D--
^ permalink raw reply [nested|flat] 48+ messages in thread
* [PATCH v2 1/1] Fix a corner-case in COPY FROM backslash processing.
@ 2021-02-04 19:36 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 48+ messages in thread
From: Heikki Linnakangas @ 2021-02-04 19:36 UTC (permalink / raw)
If a multi-byte character is escaped with a backslash in TEXT mode input,
we didn't always treat the escape correctly. If:
- a multi-byte character is escaped with a backslash, and
- the second byte of the character is 0x5C, i.e. the ASCII code of a
backslash (\), and
- the next character is a dot (.), then
CopyReadLineText function would incorrectly interpret the sequence as an
end-of-copy marker (\.). This can only happen in encodings that can
"embed" ascii characters as the second byte. One example of such sequence
is '\x5ca45c2e666f6f' in Big5 encoding. If you put that in a file, and
load it with COPY FROM, you'd incorrectly get an "end-of-copy marker
corrupt" error.
Backpatch to all supported versions.
Reviewed-by: John Naylor, Kyotaro Horiguchi
Discussion: https://www.postgresql.org/message-id/a897f84f-8dca-8798-3139-07da5bb38728%40iki.fi
---
src/backend/commands/copyfromparse.c | 10 +++++++++-
1 file changed, 9 insertions(+), 1 deletion(-)
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index b843d315b1..315b16fd7a 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -1084,7 +1084,7 @@ CopyReadLineText(CopyFromState cstate)
break;
}
else if (!cstate->opts.csv_mode)
-
+ {
/*
* If we are here, it means we found a backslash followed by
* something other than a period. In non-CSV mode, anything
@@ -1095,8 +1095,16 @@ CopyReadLineText(CopyFromState cstate)
* backslashes are not special, so we want to process the
* character after the backslash just like a normal character,
* so we don't increment in those cases.
+ *
+ * Set 'c' to skip whole character correctly in multi-byte
+ * encodings. If we don't have the whole character in the
+ * buffer yet, we might loop back to process it, after all,
+ * but that's OK because multi-byte characters cannot have any
+ * special meaning.
*/
raw_buf_ptr++;
+ c = c2;
+ }
}
/*
--
2.30.0
--------------CFC80B40DFDD66DF067F288D--
^ permalink raw reply [nested|flat] 48+ messages in thread
* [PATCH v2 1/1] Fix a corner-case in COPY FROM backslash processing.
@ 2021-02-04 19:36 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 48+ messages in thread
From: Heikki Linnakangas @ 2021-02-04 19:36 UTC (permalink / raw)
If a multi-byte character is escaped with a backslash in TEXT mode input,
we didn't always treat the escape correctly. If:
- a multi-byte character is escaped with a backslash, and
- the second byte of the character is 0x5C, i.e. the ASCII code of a
backslash (\), and
- the next character is a dot (.), then
CopyReadLineText function would incorrectly interpret the sequence as an
end-of-copy marker (\.). This can only happen in encodings that can
"embed" ascii characters as the second byte. One example of such sequence
is '\x5ca45c2e666f6f' in Big5 encoding. If you put that in a file, and
load it with COPY FROM, you'd incorrectly get an "end-of-copy marker
corrupt" error.
Backpatch to all supported versions.
Reviewed-by: John Naylor, Kyotaro Horiguchi
Discussion: https://www.postgresql.org/message-id/a897f84f-8dca-8798-3139-07da5bb38728%40iki.fi
---
src/backend/commands/copyfromparse.c | 10 +++++++++-
1 file changed, 9 insertions(+), 1 deletion(-)
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index b843d315b1..315b16fd7a 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -1084,7 +1084,7 @@ CopyReadLineText(CopyFromState cstate)
break;
}
else if (!cstate->opts.csv_mode)
-
+ {
/*
* If we are here, it means we found a backslash followed by
* something other than a period. In non-CSV mode, anything
@@ -1095,8 +1095,16 @@ CopyReadLineText(CopyFromState cstate)
* backslashes are not special, so we want to process the
* character after the backslash just like a normal character,
* so we don't increment in those cases.
+ *
+ * Set 'c' to skip whole character correctly in multi-byte
+ * encodings. If we don't have the whole character in the
+ * buffer yet, we might loop back to process it, after all,
+ * but that's OK because multi-byte characters cannot have any
+ * special meaning.
*/
raw_buf_ptr++;
+ c = c2;
+ }
}
/*
--
2.30.0
--------------CFC80B40DFDD66DF067F288D--
^ permalink raw reply [nested|flat] 48+ messages in thread
* [PATCH v2 1/1] Fix a corner-case in COPY FROM backslash processing.
@ 2021-02-04 19:36 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 48+ messages in thread
From: Heikki Linnakangas @ 2021-02-04 19:36 UTC (permalink / raw)
If a multi-byte character is escaped with a backslash in TEXT mode input,
we didn't always treat the escape correctly. If:
- a multi-byte character is escaped with a backslash, and
- the second byte of the character is 0x5C, i.e. the ASCII code of a
backslash (\), and
- the next character is a dot (.), then
CopyReadLineText function would incorrectly interpret the sequence as an
end-of-copy marker (\.). This can only happen in encodings that can
"embed" ascii characters as the second byte. One example of such sequence
is '\x5ca45c2e666f6f' in Big5 encoding. If you put that in a file, and
load it with COPY FROM, you'd incorrectly get an "end-of-copy marker
corrupt" error.
Backpatch to all supported versions.
Reviewed-by: John Naylor, Kyotaro Horiguchi
Discussion: https://www.postgresql.org/message-id/a897f84f-8dca-8798-3139-07da5bb38728%40iki.fi
---
src/backend/commands/copyfromparse.c | 10 +++++++++-
1 file changed, 9 insertions(+), 1 deletion(-)
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index b843d315b1..315b16fd7a 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -1084,7 +1084,7 @@ CopyReadLineText(CopyFromState cstate)
break;
}
else if (!cstate->opts.csv_mode)
-
+ {
/*
* If we are here, it means we found a backslash followed by
* something other than a period. In non-CSV mode, anything
@@ -1095,8 +1095,16 @@ CopyReadLineText(CopyFromState cstate)
* backslashes are not special, so we want to process the
* character after the backslash just like a normal character,
* so we don't increment in those cases.
+ *
+ * Set 'c' to skip whole character correctly in multi-byte
+ * encodings. If we don't have the whole character in the
+ * buffer yet, we might loop back to process it, after all,
+ * but that's OK because multi-byte characters cannot have any
+ * special meaning.
*/
raw_buf_ptr++;
+ c = c2;
+ }
}
/*
--
2.30.0
--------------CFC80B40DFDD66DF067F288D--
^ permalink raw reply [nested|flat] 48+ messages in thread
* [PATCH v2 1/1] Fix a corner-case in COPY FROM backslash processing.
@ 2021-02-04 19:36 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 48+ messages in thread
From: Heikki Linnakangas @ 2021-02-04 19:36 UTC (permalink / raw)
If a multi-byte character is escaped with a backslash in TEXT mode input,
we didn't always treat the escape correctly. If:
- a multi-byte character is escaped with a backslash, and
- the second byte of the character is 0x5C, i.e. the ASCII code of a
backslash (\), and
- the next character is a dot (.), then
CopyReadLineText function would incorrectly interpret the sequence as an
end-of-copy marker (\.). This can only happen in encodings that can
"embed" ascii characters as the second byte. One example of such sequence
is '\x5ca45c2e666f6f' in Big5 encoding. If you put that in a file, and
load it with COPY FROM, you'd incorrectly get an "end-of-copy marker
corrupt" error.
Backpatch to all supported versions.
Reviewed-by: John Naylor, Kyotaro Horiguchi
Discussion: https://www.postgresql.org/message-id/a897f84f-8dca-8798-3139-07da5bb38728%40iki.fi
---
src/backend/commands/copyfromparse.c | 10 +++++++++-
1 file changed, 9 insertions(+), 1 deletion(-)
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index b843d315b1..315b16fd7a 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -1084,7 +1084,7 @@ CopyReadLineText(CopyFromState cstate)
break;
}
else if (!cstate->opts.csv_mode)
-
+ {
/*
* If we are here, it means we found a backslash followed by
* something other than a period. In non-CSV mode, anything
@@ -1095,8 +1095,16 @@ CopyReadLineText(CopyFromState cstate)
* backslashes are not special, so we want to process the
* character after the backslash just like a normal character,
* so we don't increment in those cases.
+ *
+ * Set 'c' to skip whole character correctly in multi-byte
+ * encodings. If we don't have the whole character in the
+ * buffer yet, we might loop back to process it, after all,
+ * but that's OK because multi-byte characters cannot have any
+ * special meaning.
*/
raw_buf_ptr++;
+ c = c2;
+ }
}
/*
--
2.30.0
--------------CFC80B40DFDD66DF067F288D--
^ permalink raw reply [nested|flat] 48+ messages in thread
* [PATCH v2 1/1] Fix a corner-case in COPY FROM backslash processing.
@ 2021-02-04 19:36 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 48+ messages in thread
From: Heikki Linnakangas @ 2021-02-04 19:36 UTC (permalink / raw)
If a multi-byte character is escaped with a backslash in TEXT mode input,
we didn't always treat the escape correctly. If:
- a multi-byte character is escaped with a backslash, and
- the second byte of the character is 0x5C, i.e. the ASCII code of a
backslash (\), and
- the next character is a dot (.), then
CopyReadLineText function would incorrectly interpret the sequence as an
end-of-copy marker (\.). This can only happen in encodings that can
"embed" ascii characters as the second byte. One example of such sequence
is '\x5ca45c2e666f6f' in Big5 encoding. If you put that in a file, and
load it with COPY FROM, you'd incorrectly get an "end-of-copy marker
corrupt" error.
Backpatch to all supported versions.
Reviewed-by: John Naylor, Kyotaro Horiguchi
Discussion: https://www.postgresql.org/message-id/a897f84f-8dca-8798-3139-07da5bb38728%40iki.fi
---
src/backend/commands/copyfromparse.c | 10 +++++++++-
1 file changed, 9 insertions(+), 1 deletion(-)
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index b843d315b1..315b16fd7a 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -1084,7 +1084,7 @@ CopyReadLineText(CopyFromState cstate)
break;
}
else if (!cstate->opts.csv_mode)
-
+ {
/*
* If we are here, it means we found a backslash followed by
* something other than a period. In non-CSV mode, anything
@@ -1095,8 +1095,16 @@ CopyReadLineText(CopyFromState cstate)
* backslashes are not special, so we want to process the
* character after the backslash just like a normal character,
* so we don't increment in those cases.
+ *
+ * Set 'c' to skip whole character correctly in multi-byte
+ * encodings. If we don't have the whole character in the
+ * buffer yet, we might loop back to process it, after all,
+ * but that's OK because multi-byte characters cannot have any
+ * special meaning.
*/
raw_buf_ptr++;
+ c = c2;
+ }
}
/*
--
2.30.0
--------------CFC80B40DFDD66DF067F288D--
^ permalink raw reply [nested|flat] 48+ messages in thread
* [PATCH v2 1/1] Fix a corner-case in COPY FROM backslash processing.
@ 2021-02-04 19:36 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 48+ messages in thread
From: Heikki Linnakangas @ 2021-02-04 19:36 UTC (permalink / raw)
If a multi-byte character is escaped with a backslash in TEXT mode input,
we didn't always treat the escape correctly. If:
- a multi-byte character is escaped with a backslash, and
- the second byte of the character is 0x5C, i.e. the ASCII code of a
backslash (\), and
- the next character is a dot (.), then
CopyReadLineText function would incorrectly interpret the sequence as an
end-of-copy marker (\.). This can only happen in encodings that can
"embed" ascii characters as the second byte. One example of such sequence
is '\x5ca45c2e666f6f' in Big5 encoding. If you put that in a file, and
load it with COPY FROM, you'd incorrectly get an "end-of-copy marker
corrupt" error.
Backpatch to all supported versions.
Reviewed-by: John Naylor, Kyotaro Horiguchi
Discussion: https://www.postgresql.org/message-id/a897f84f-8dca-8798-3139-07da5bb38728%40iki.fi
---
src/backend/commands/copyfromparse.c | 10 +++++++++-
1 file changed, 9 insertions(+), 1 deletion(-)
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index b843d315b1..315b16fd7a 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -1084,7 +1084,7 @@ CopyReadLineText(CopyFromState cstate)
break;
}
else if (!cstate->opts.csv_mode)
-
+ {
/*
* If we are here, it means we found a backslash followed by
* something other than a period. In non-CSV mode, anything
@@ -1095,8 +1095,16 @@ CopyReadLineText(CopyFromState cstate)
* backslashes are not special, so we want to process the
* character after the backslash just like a normal character,
* so we don't increment in those cases.
+ *
+ * Set 'c' to skip whole character correctly in multi-byte
+ * encodings. If we don't have the whole character in the
+ * buffer yet, we might loop back to process it, after all,
+ * but that's OK because multi-byte characters cannot have any
+ * special meaning.
*/
raw_buf_ptr++;
+ c = c2;
+ }
}
/*
--
2.30.0
--------------CFC80B40DFDD66DF067F288D--
^ permalink raw reply [nested|flat] 48+ messages in thread
* [PATCH v2 1/1] Fix a corner-case in COPY FROM backslash processing.
@ 2021-02-04 19:36 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 48+ messages in thread
From: Heikki Linnakangas @ 2021-02-04 19:36 UTC (permalink / raw)
If a multi-byte character is escaped with a backslash in TEXT mode input,
we didn't always treat the escape correctly. If:
- a multi-byte character is escaped with a backslash, and
- the second byte of the character is 0x5C, i.e. the ASCII code of a
backslash (\), and
- the next character is a dot (.), then
CopyReadLineText function would incorrectly interpret the sequence as an
end-of-copy marker (\.). This can only happen in encodings that can
"embed" ascii characters as the second byte. One example of such sequence
is '\x5ca45c2e666f6f' in Big5 encoding. If you put that in a file, and
load it with COPY FROM, you'd incorrectly get an "end-of-copy marker
corrupt" error.
Backpatch to all supported versions.
Reviewed-by: John Naylor, Kyotaro Horiguchi
Discussion: https://www.postgresql.org/message-id/a897f84f-8dca-8798-3139-07da5bb38728%40iki.fi
---
src/backend/commands/copyfromparse.c | 10 +++++++++-
1 file changed, 9 insertions(+), 1 deletion(-)
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index b843d315b1..315b16fd7a 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -1084,7 +1084,7 @@ CopyReadLineText(CopyFromState cstate)
break;
}
else if (!cstate->opts.csv_mode)
-
+ {
/*
* If we are here, it means we found a backslash followed by
* something other than a period. In non-CSV mode, anything
@@ -1095,8 +1095,16 @@ CopyReadLineText(CopyFromState cstate)
* backslashes are not special, so we want to process the
* character after the backslash just like a normal character,
* so we don't increment in those cases.
+ *
+ * Set 'c' to skip whole character correctly in multi-byte
+ * encodings. If we don't have the whole character in the
+ * buffer yet, we might loop back to process it, after all,
+ * but that's OK because multi-byte characters cannot have any
+ * special meaning.
*/
raw_buf_ptr++;
+ c = c2;
+ }
}
/*
--
2.30.0
--------------CFC80B40DFDD66DF067F288D--
^ permalink raw reply [nested|flat] 48+ messages in thread
* [PATCH v2 1/1] Fix a corner-case in COPY FROM backslash processing.
@ 2021-02-04 19:36 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 48+ messages in thread
From: Heikki Linnakangas @ 2021-02-04 19:36 UTC (permalink / raw)
If a multi-byte character is escaped with a backslash in TEXT mode input,
we didn't always treat the escape correctly. If:
- a multi-byte character is escaped with a backslash, and
- the second byte of the character is 0x5C, i.e. the ASCII code of a
backslash (\), and
- the next character is a dot (.), then
CopyReadLineText function would incorrectly interpret the sequence as an
end-of-copy marker (\.). This can only happen in encodings that can
"embed" ascii characters as the second byte. One example of such sequence
is '\x5ca45c2e666f6f' in Big5 encoding. If you put that in a file, and
load it with COPY FROM, you'd incorrectly get an "end-of-copy marker
corrupt" error.
Backpatch to all supported versions.
Reviewed-by: John Naylor, Kyotaro Horiguchi
Discussion: https://www.postgresql.org/message-id/a897f84f-8dca-8798-3139-07da5bb38728%40iki.fi
---
src/backend/commands/copyfromparse.c | 10 +++++++++-
1 file changed, 9 insertions(+), 1 deletion(-)
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index b843d315b1..315b16fd7a 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -1084,7 +1084,7 @@ CopyReadLineText(CopyFromState cstate)
break;
}
else if (!cstate->opts.csv_mode)
-
+ {
/*
* If we are here, it means we found a backslash followed by
* something other than a period. In non-CSV mode, anything
@@ -1095,8 +1095,16 @@ CopyReadLineText(CopyFromState cstate)
* backslashes are not special, so we want to process the
* character after the backslash just like a normal character,
* so we don't increment in those cases.
+ *
+ * Set 'c' to skip whole character correctly in multi-byte
+ * encodings. If we don't have the whole character in the
+ * buffer yet, we might loop back to process it, after all,
+ * but that's OK because multi-byte characters cannot have any
+ * special meaning.
*/
raw_buf_ptr++;
+ c = c2;
+ }
}
/*
--
2.30.0
--------------CFC80B40DFDD66DF067F288D--
^ permalink raw reply [nested|flat] 48+ messages in thread
* [PATCH v2 1/1] Fix a corner-case in COPY FROM backslash processing.
@ 2021-02-04 19:36 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 48+ messages in thread
From: Heikki Linnakangas @ 2021-02-04 19:36 UTC (permalink / raw)
If a multi-byte character is escaped with a backslash in TEXT mode input,
we didn't always treat the escape correctly. If:
- a multi-byte character is escaped with a backslash, and
- the second byte of the character is 0x5C, i.e. the ASCII code of a
backslash (\), and
- the next character is a dot (.), then
CopyReadLineText function would incorrectly interpret the sequence as an
end-of-copy marker (\.). This can only happen in encodings that can
"embed" ascii characters as the second byte. One example of such sequence
is '\x5ca45c2e666f6f' in Big5 encoding. If you put that in a file, and
load it with COPY FROM, you'd incorrectly get an "end-of-copy marker
corrupt" error.
Backpatch to all supported versions.
Reviewed-by: John Naylor, Kyotaro Horiguchi
Discussion: https://www.postgresql.org/message-id/a897f84f-8dca-8798-3139-07da5bb38728%40iki.fi
---
src/backend/commands/copyfromparse.c | 10 +++++++++-
1 file changed, 9 insertions(+), 1 deletion(-)
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index b843d315b1..315b16fd7a 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -1084,7 +1084,7 @@ CopyReadLineText(CopyFromState cstate)
break;
}
else if (!cstate->opts.csv_mode)
-
+ {
/*
* If we are here, it means we found a backslash followed by
* something other than a period. In non-CSV mode, anything
@@ -1095,8 +1095,16 @@ CopyReadLineText(CopyFromState cstate)
* backslashes are not special, so we want to process the
* character after the backslash just like a normal character,
* so we don't increment in those cases.
+ *
+ * Set 'c' to skip whole character correctly in multi-byte
+ * encodings. If we don't have the whole character in the
+ * buffer yet, we might loop back to process it, after all,
+ * but that's OK because multi-byte characters cannot have any
+ * special meaning.
*/
raw_buf_ptr++;
+ c = c2;
+ }
}
/*
--
2.30.0
--------------CFC80B40DFDD66DF067F288D--
^ permalink raw reply [nested|flat] 48+ messages in thread
* [PATCH v2 1/1] Fix a corner-case in COPY FROM backslash processing.
@ 2021-02-04 19:36 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 48+ messages in thread
From: Heikki Linnakangas @ 2021-02-04 19:36 UTC (permalink / raw)
If a multi-byte character is escaped with a backslash in TEXT mode input,
we didn't always treat the escape correctly. If:
- a multi-byte character is escaped with a backslash, and
- the second byte of the character is 0x5C, i.e. the ASCII code of a
backslash (\), and
- the next character is a dot (.), then
CopyReadLineText function would incorrectly interpret the sequence as an
end-of-copy marker (\.). This can only happen in encodings that can
"embed" ascii characters as the second byte. One example of such sequence
is '\x5ca45c2e666f6f' in Big5 encoding. If you put that in a file, and
load it with COPY FROM, you'd incorrectly get an "end-of-copy marker
corrupt" error.
Backpatch to all supported versions.
Reviewed-by: John Naylor, Kyotaro Horiguchi
Discussion: https://www.postgresql.org/message-id/a897f84f-8dca-8798-3139-07da5bb38728%40iki.fi
---
src/backend/commands/copyfromparse.c | 10 +++++++++-
1 file changed, 9 insertions(+), 1 deletion(-)
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index b843d315b1..315b16fd7a 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -1084,7 +1084,7 @@ CopyReadLineText(CopyFromState cstate)
break;
}
else if (!cstate->opts.csv_mode)
-
+ {
/*
* If we are here, it means we found a backslash followed by
* something other than a period. In non-CSV mode, anything
@@ -1095,8 +1095,16 @@ CopyReadLineText(CopyFromState cstate)
* backslashes are not special, so we want to process the
* character after the backslash just like a normal character,
* so we don't increment in those cases.
+ *
+ * Set 'c' to skip whole character correctly in multi-byte
+ * encodings. If we don't have the whole character in the
+ * buffer yet, we might loop back to process it, after all,
+ * but that's OK because multi-byte characters cannot have any
+ * special meaning.
*/
raw_buf_ptr++;
+ c = c2;
+ }
}
/*
--
2.30.0
--------------CFC80B40DFDD66DF067F288D--
^ permalink raw reply [nested|flat] 48+ messages in thread
* [PATCH v2 1/1] Fix a corner-case in COPY FROM backslash processing.
@ 2021-02-04 19:36 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 48+ messages in thread
From: Heikki Linnakangas @ 2021-02-04 19:36 UTC (permalink / raw)
If a multi-byte character is escaped with a backslash in TEXT mode input,
we didn't always treat the escape correctly. If:
- a multi-byte character is escaped with a backslash, and
- the second byte of the character is 0x5C, i.e. the ASCII code of a
backslash (\), and
- the next character is a dot (.), then
CopyReadLineText function would incorrectly interpret the sequence as an
end-of-copy marker (\.). This can only happen in encodings that can
"embed" ascii characters as the second byte. One example of such sequence
is '\x5ca45c2e666f6f' in Big5 encoding. If you put that in a file, and
load it with COPY FROM, you'd incorrectly get an "end-of-copy marker
corrupt" error.
Backpatch to all supported versions.
Reviewed-by: John Naylor, Kyotaro Horiguchi
Discussion: https://www.postgresql.org/message-id/a897f84f-8dca-8798-3139-07da5bb38728%40iki.fi
---
src/backend/commands/copyfromparse.c | 10 +++++++++-
1 file changed, 9 insertions(+), 1 deletion(-)
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index b843d315b1..315b16fd7a 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -1084,7 +1084,7 @@ CopyReadLineText(CopyFromState cstate)
break;
}
else if (!cstate->opts.csv_mode)
-
+ {
/*
* If we are here, it means we found a backslash followed by
* something other than a period. In non-CSV mode, anything
@@ -1095,8 +1095,16 @@ CopyReadLineText(CopyFromState cstate)
* backslashes are not special, so we want to process the
* character after the backslash just like a normal character,
* so we don't increment in those cases.
+ *
+ * Set 'c' to skip whole character correctly in multi-byte
+ * encodings. If we don't have the whole character in the
+ * buffer yet, we might loop back to process it, after all,
+ * but that's OK because multi-byte characters cannot have any
+ * special meaning.
*/
raw_buf_ptr++;
+ c = c2;
+ }
}
/*
--
2.30.0
--------------CFC80B40DFDD66DF067F288D--
^ permalink raw reply [nested|flat] 48+ messages in thread
* [PATCH v2 1/1] Fix a corner-case in COPY FROM backslash processing.
@ 2021-02-04 19:36 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 48+ messages in thread
From: Heikki Linnakangas @ 2021-02-04 19:36 UTC (permalink / raw)
If a multi-byte character is escaped with a backslash in TEXT mode input,
we didn't always treat the escape correctly. If:
- a multi-byte character is escaped with a backslash, and
- the second byte of the character is 0x5C, i.e. the ASCII code of a
backslash (\), and
- the next character is a dot (.), then
CopyReadLineText function would incorrectly interpret the sequence as an
end-of-copy marker (\.). This can only happen in encodings that can
"embed" ascii characters as the second byte. One example of such sequence
is '\x5ca45c2e666f6f' in Big5 encoding. If you put that in a file, and
load it with COPY FROM, you'd incorrectly get an "end-of-copy marker
corrupt" error.
Backpatch to all supported versions.
Reviewed-by: John Naylor, Kyotaro Horiguchi
Discussion: https://www.postgresql.org/message-id/a897f84f-8dca-8798-3139-07da5bb38728%40iki.fi
---
src/backend/commands/copyfromparse.c | 10 +++++++++-
1 file changed, 9 insertions(+), 1 deletion(-)
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index b843d315b1..315b16fd7a 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -1084,7 +1084,7 @@ CopyReadLineText(CopyFromState cstate)
break;
}
else if (!cstate->opts.csv_mode)
-
+ {
/*
* If we are here, it means we found a backslash followed by
* something other than a period. In non-CSV mode, anything
@@ -1095,8 +1095,16 @@ CopyReadLineText(CopyFromState cstate)
* backslashes are not special, so we want to process the
* character after the backslash just like a normal character,
* so we don't increment in those cases.
+ *
+ * Set 'c' to skip whole character correctly in multi-byte
+ * encodings. If we don't have the whole character in the
+ * buffer yet, we might loop back to process it, after all,
+ * but that's OK because multi-byte characters cannot have any
+ * special meaning.
*/
raw_buf_ptr++;
+ c = c2;
+ }
}
/*
--
2.30.0
--------------CFC80B40DFDD66DF067F288D--
^ permalink raw reply [nested|flat] 48+ messages in thread
* [PATCH v2 1/1] Fix a corner-case in COPY FROM backslash processing.
@ 2021-02-04 19:36 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 48+ messages in thread
From: Heikki Linnakangas @ 2021-02-04 19:36 UTC (permalink / raw)
If a multi-byte character is escaped with a backslash in TEXT mode input,
we didn't always treat the escape correctly. If:
- a multi-byte character is escaped with a backslash, and
- the second byte of the character is 0x5C, i.e. the ASCII code of a
backslash (\), and
- the next character is a dot (.), then
CopyReadLineText function would incorrectly interpret the sequence as an
end-of-copy marker (\.). This can only happen in encodings that can
"embed" ascii characters as the second byte. One example of such sequence
is '\x5ca45c2e666f6f' in Big5 encoding. If you put that in a file, and
load it with COPY FROM, you'd incorrectly get an "end-of-copy marker
corrupt" error.
Backpatch to all supported versions.
Reviewed-by: John Naylor, Kyotaro Horiguchi
Discussion: https://www.postgresql.org/message-id/a897f84f-8dca-8798-3139-07da5bb38728%40iki.fi
---
src/backend/commands/copyfromparse.c | 10 +++++++++-
1 file changed, 9 insertions(+), 1 deletion(-)
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index b843d315b1..315b16fd7a 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -1084,7 +1084,7 @@ CopyReadLineText(CopyFromState cstate)
break;
}
else if (!cstate->opts.csv_mode)
-
+ {
/*
* If we are here, it means we found a backslash followed by
* something other than a period. In non-CSV mode, anything
@@ -1095,8 +1095,16 @@ CopyReadLineText(CopyFromState cstate)
* backslashes are not special, so we want to process the
* character after the backslash just like a normal character,
* so we don't increment in those cases.
+ *
+ * Set 'c' to skip whole character correctly in multi-byte
+ * encodings. If we don't have the whole character in the
+ * buffer yet, we might loop back to process it, after all,
+ * but that's OK because multi-byte characters cannot have any
+ * special meaning.
*/
raw_buf_ptr++;
+ c = c2;
+ }
}
/*
--
2.30.0
--------------CFC80B40DFDD66DF067F288D--
^ permalink raw reply [nested|flat] 48+ messages in thread
* [PATCH v2 1/1] Fix a corner-case in COPY FROM backslash processing.
@ 2021-02-04 19:36 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 48+ messages in thread
From: Heikki Linnakangas @ 2021-02-04 19:36 UTC (permalink / raw)
If a multi-byte character is escaped with a backslash in TEXT mode input,
we didn't always treat the escape correctly. If:
- a multi-byte character is escaped with a backslash, and
- the second byte of the character is 0x5C, i.e. the ASCII code of a
backslash (\), and
- the next character is a dot (.), then
CopyReadLineText function would incorrectly interpret the sequence as an
end-of-copy marker (\.). This can only happen in encodings that can
"embed" ascii characters as the second byte. One example of such sequence
is '\x5ca45c2e666f6f' in Big5 encoding. If you put that in a file, and
load it with COPY FROM, you'd incorrectly get an "end-of-copy marker
corrupt" error.
Backpatch to all supported versions.
Reviewed-by: John Naylor, Kyotaro Horiguchi
Discussion: https://www.postgresql.org/message-id/a897f84f-8dca-8798-3139-07da5bb38728%40iki.fi
---
src/backend/commands/copyfromparse.c | 10 +++++++++-
1 file changed, 9 insertions(+), 1 deletion(-)
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index b843d315b1..315b16fd7a 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -1084,7 +1084,7 @@ CopyReadLineText(CopyFromState cstate)
break;
}
else if (!cstate->opts.csv_mode)
-
+ {
/*
* If we are here, it means we found a backslash followed by
* something other than a period. In non-CSV mode, anything
@@ -1095,8 +1095,16 @@ CopyReadLineText(CopyFromState cstate)
* backslashes are not special, so we want to process the
* character after the backslash just like a normal character,
* so we don't increment in those cases.
+ *
+ * Set 'c' to skip whole character correctly in multi-byte
+ * encodings. If we don't have the whole character in the
+ * buffer yet, we might loop back to process it, after all,
+ * but that's OK because multi-byte characters cannot have any
+ * special meaning.
*/
raw_buf_ptr++;
+ c = c2;
+ }
}
/*
--
2.30.0
--------------CFC80B40DFDD66DF067F288D--
^ permalink raw reply [nested|flat] 48+ messages in thread
* [PATCH v2 1/1] Fix a corner-case in COPY FROM backslash processing.
@ 2021-02-04 19:36 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 48+ messages in thread
From: Heikki Linnakangas @ 2021-02-04 19:36 UTC (permalink / raw)
If a multi-byte character is escaped with a backslash in TEXT mode input,
we didn't always treat the escape correctly. If:
- a multi-byte character is escaped with a backslash, and
- the second byte of the character is 0x5C, i.e. the ASCII code of a
backslash (\), and
- the next character is a dot (.), then
CopyReadLineText function would incorrectly interpret the sequence as an
end-of-copy marker (\.). This can only happen in encodings that can
"embed" ascii characters as the second byte. One example of such sequence
is '\x5ca45c2e666f6f' in Big5 encoding. If you put that in a file, and
load it with COPY FROM, you'd incorrectly get an "end-of-copy marker
corrupt" error.
Backpatch to all supported versions.
Reviewed-by: John Naylor, Kyotaro Horiguchi
Discussion: https://www.postgresql.org/message-id/a897f84f-8dca-8798-3139-07da5bb38728%40iki.fi
---
src/backend/commands/copyfromparse.c | 10 +++++++++-
1 file changed, 9 insertions(+), 1 deletion(-)
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index b843d315b1..315b16fd7a 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -1084,7 +1084,7 @@ CopyReadLineText(CopyFromState cstate)
break;
}
else if (!cstate->opts.csv_mode)
-
+ {
/*
* If we are here, it means we found a backslash followed by
* something other than a period. In non-CSV mode, anything
@@ -1095,8 +1095,16 @@ CopyReadLineText(CopyFromState cstate)
* backslashes are not special, so we want to process the
* character after the backslash just like a normal character,
* so we don't increment in those cases.
+ *
+ * Set 'c' to skip whole character correctly in multi-byte
+ * encodings. If we don't have the whole character in the
+ * buffer yet, we might loop back to process it, after all,
+ * but that's OK because multi-byte characters cannot have any
+ * special meaning.
*/
raw_buf_ptr++;
+ c = c2;
+ }
}
/*
--
2.30.0
--------------CFC80B40DFDD66DF067F288D--
^ permalink raw reply [nested|flat] 48+ messages in thread
* [PATCH v2 1/1] Fix a corner-case in COPY FROM backslash processing.
@ 2021-02-04 19:36 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 48+ messages in thread
From: Heikki Linnakangas @ 2021-02-04 19:36 UTC (permalink / raw)
If a multi-byte character is escaped with a backslash in TEXT mode input,
we didn't always treat the escape correctly. If:
- a multi-byte character is escaped with a backslash, and
- the second byte of the character is 0x5C, i.e. the ASCII code of a
backslash (\), and
- the next character is a dot (.), then
CopyReadLineText function would incorrectly interpret the sequence as an
end-of-copy marker (\.). This can only happen in encodings that can
"embed" ascii characters as the second byte. One example of such sequence
is '\x5ca45c2e666f6f' in Big5 encoding. If you put that in a file, and
load it with COPY FROM, you'd incorrectly get an "end-of-copy marker
corrupt" error.
Backpatch to all supported versions.
Reviewed-by: John Naylor, Kyotaro Horiguchi
Discussion: https://www.postgresql.org/message-id/a897f84f-8dca-8798-3139-07da5bb38728%40iki.fi
---
src/backend/commands/copyfromparse.c | 10 +++++++++-
1 file changed, 9 insertions(+), 1 deletion(-)
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index b843d315b1..315b16fd7a 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -1084,7 +1084,7 @@ CopyReadLineText(CopyFromState cstate)
break;
}
else if (!cstate->opts.csv_mode)
-
+ {
/*
* If we are here, it means we found a backslash followed by
* something other than a period. In non-CSV mode, anything
@@ -1095,8 +1095,16 @@ CopyReadLineText(CopyFromState cstate)
* backslashes are not special, so we want to process the
* character after the backslash just like a normal character,
* so we don't increment in those cases.
+ *
+ * Set 'c' to skip whole character correctly in multi-byte
+ * encodings. If we don't have the whole character in the
+ * buffer yet, we might loop back to process it, after all,
+ * but that's OK because multi-byte characters cannot have any
+ * special meaning.
*/
raw_buf_ptr++;
+ c = c2;
+ }
}
/*
--
2.30.0
--------------CFC80B40DFDD66DF067F288D--
^ permalink raw reply [nested|flat] 48+ messages in thread
* Re: Adjust pg_stat_get_lock() prorows to match lock types
@ 2026-06-06 03:30 =?utf-8?B?Wml6aHVhbkxpdSBYLU1BTg==?= <[email protected]>
1 sibling, 0 replies; 48+ messages in thread
From: =?utf-8?B?Wml6aHVhbkxpdSBYLU1BTg==?= @ 2026-06-06 03:30 UTC (permalink / raw)
To: =?utf-8?B?Q2hhbyBMaQ==?= <[email protected]>; =?utf-8?B?UG9zdGdyZXMgaGFja2Vycw==?= <[email protected]>
>From: Chao Li <[email protected]>
>Date: 2026-05-15 16:34
>To: Postgres hackers <[email protected]>
>Subject: Re: Adjust pg_stat_get_lock() prorows to match lock types
>
>
>
>> On May 4, 2026, at 10:23, Chao Li <[email protected]> wrote:
>>
>> Hi,
>>
>> I read the code of pg_stat_lock() and played a bit with it. I happened to notice one thing: the function always returns 12 rows, but the planner estimates 10 rows:
>>
>> ```
>> evantest=# EXPLAIN ANALYZE SELECT * FROM pg_catalog.pg_stat_lock;
>> QUERY PLAN
>> -----------------------------------------------------------------------------------------------------------------------
>> Function Scan on pg_stat_get_lock l (cost=0.00..0.10 rows=10 width=64) (actual time=0.067..0.071 rows=12.00 loops=1)
>> Planning Time: 0.121 ms
>> Execution Time: 0.126 ms
>> (3 rows)
>> ```
>>
>> Then I found that, in pg_proc.dat, the function's prorows is defined as 10. Since the function returns one row per lock type, and lock types are not something that change frequently, I think it is better to give the planner a more accurate row count. After changing prorows to 12, the plan looks like this:
>>
>> ```
>> evantest=# EXPLAIN ANALYZE SELECT * FROM pg_catalog.pg_stat_lock;
>> QUERY PLAN
>> -----------------------------------------------------------------------------------------------------------------------
>> Function Scan on pg_stat_get_lock l (cost=0.00..0.12 rows=12 width=64) (actual time=0.134..0.138 rows=12.00 loops=1)
>> Planning:
>> Buffers: shared hit=13
>> Planning Time: 0.313 ms
>> Execution Time: 0.228 ms
>> (5 rows)
>> ```
>>
>> While there, I also made two small tweaks to two function comments in pgstat_lock.c. If those are not considered worth changing, I am okay with removing them from the patch.
>>
>> Please see the attached patch for details.
>>
>> Best regards,
>> --
>> Chao Li (Evan)
>> HighGo Software Co., Ltd.
>> https://www.highgo.com/
>>
>>
>>
>>
>> <v1-0001-Adjust-pg_stat_get_lock-row-estimate-and-comments.patch>
>
>Rebased.
>
>Best regards,
>--
>Chao Li (Evan)
>HighGo Software Co., Ltd.
>https://www.highgo.com/
Hi Chao,
After testing with GDB, I found that before your patch, the catalog data of pg_stat_get_lock shows (proowner = 10):
{oid = 6509, proname = {data = "pg_stat_get_lock", '\000' <repeats 47 times>}, pronamespace = 11, proowner = 10, prolang = 12, procost = 1, prorows = 10, provariadic = 0, prosupport = 0, prokind = 102 'f', prosecdef = false, proleakproof = false, proisstrict = true, proretset = true, provolatile = 118 'v', proparallel = 114 'r', pronargs = 0, pronargdefaults = 0, prorettype = 2249, proargtypes = {vl_len_ = 96, ndim = 1, dataoffset = 0, elemtype = 26, dim1 = 0, lbound1 = 0, values = 0x7f09190754e0}}
`
With your v2 patch applied, the data becomes (proowner = 12):
{oid = 6509, proname = {data = "pg_stat_get_lock", '\000' <repeats 47 times>}, pronamespace = 11, proowner = 10, prolang = 12, procost = 1,
prorows = 12, provariadic = 0, prosupport = 0, prokind = 102 'f', prosecdef = false, proleakproof = false, proisstrict = true, proretset = true,
provolatile = 118 'v', proparallel = 114 'r', pronargs = 0, pronargdefaults = 0, prorettype = 2249, proargtypes = {vl_len_ = 96, ndim = 1,
dataoffset = 0, elemtype = 26, dim1 = 0, lbound1 = 0, values = 0x7f013c201788}}
`
The estimated row count 12 of pg_stat_get_lock matches the number of lockable object types:
relation, extend, frozenid, page, tuple, transactionid, virtualxid, spectoken, object, userlock, advisory, and applytransaction.
This is exactly as mentioned in the documentation "Table 27.11. Wait Events of Type Lock", as shown below.
`
xman=# explain select * from pg_catalog.pg_stat_lock;
QUERY PLAN
-------------------------------------------------------------------------
Function Scan on pg_stat_get_lock l (cost=0.00..0.12 rows=12 width=64)
(1 row)
xman=# explain analyze select * from pg_catalog.pg_stat_lock;
QUERY PLAN
-----------------------------------------------------------------------------------------------------------------------
Function Scan on pg_stat_get_lock l (cost=0.00..0.12 rows=12 width=64) (actual time=0.115..0.124 rows=12.00 loops=1)
Planning Time: 4941.026 ms
Execution Time: 0.240 ms
(3 rows)
xman=# select * from pg_catalog.pg_stat_lock;
locktype | waits | wait_time | fastpath_exceeded | stats_reset
------------------+-------+-----------+-------------------+-------------------------------
relation | 0 | 0 | 0 | 2026-06-06 10:59:06.350828+08
extend | 0 | 0 | 0 | 2026-06-06 10:59:06.350828+08
frozenid | 0 | 0 | 0 | 2026-06-06 10:59:06.350828+08
page | 0 | 0 | 0 | 2026-06-06 10:59:06.350828+08
tuple | 0 | 0 | 0 | 2026-06-06 10:59:06.350828+08
transactionid | 0 | 0 | 0 | 2026-06-06 10:59:06.350828+08
virtualxid | 0 | 0 | 0 | 2026-06-06 10:59:06.350828+08
spectoken | 0 | 0 | 0 | 2026-06-06 10:59:06.350828+08
object | 0 | 0 | 0 | 2026-06-06 10:59:06.350828+08
userlock | 0 | 0 | 0 | 2026-06-06 10:59:06.350828+08
advisory | 0 | 0 | 0 | 2026-06-06 10:59:06.350828+08
applytransaction | 0 | 0 | 0 | 2026-06-06 10:59:06.350828+08
(12 rows)
`
Thanks for the patch.
regards,
--
ZizhuanLiu (X-MAN)
[email protected]
^ permalink raw reply [nested|flat] 48+ messages in thread
* Re: Adjust pg_stat_get_lock() prorows to match lock types
@ 2026-06-08 05:20 Kyotaro Horiguchi <[email protected]>
1 sibling, 2 replies; 48+ messages in thread
From: Kyotaro Horiguchi @ 2026-06-08 05:20 UTC (permalink / raw)
To: [email protected]; +Cc: [email protected]
Hello.
At Mon, 4 May 2026 10:23:47 +0800, Chao Li <[email protected]> wrote in
> I read the code of pg_stat_lock() and played a bit with it. I happened =
> to notice one thing: the function always returns 12 rows, but the =
> planner estimates 10 rows:
I'm not convinced this needs to be adjusted.
The prorows value is only a rough estimate for the planner, not an
exact row count. For example, pg_show_all_settings() has prorows =
1000, while it returns substantially fewer rows in my build. That
suggests these values are intended to provide a reasonable estimate
rather than an exact count.
Given that, I don't think a difference between 10 and 12 rows is worth
fixing. Keeping prorows synchronized with the exact number of lock
types would only add maintenance overhead whenever a new lock type is
introduced.
If the current estimate is shown to cause a planning problem, I'd
rather address that problem directly than try to keep prorows exactly
synchronized with the number of lock types.
This discussion also made me wonder whether the documentation could be
a bit more explicit that prorows is intended as a planner estimate
rather than an exact row count.
> Estimated number of result rows (zero if not proretset)
Regards.
--
Kyotaro Horiguchi
NTT Open Source Software Center
^ permalink raw reply [nested|flat] 48+ messages in thread
* Re: Adjust pg_stat_get_lock() prorows to match lock types
@ 2026-06-08 05:45 Michael Paquier <[email protected]>
parent: Kyotaro Horiguchi <[email protected]>
1 sibling, 0 replies; 48+ messages in thread
From: Michael Paquier @ 2026-06-08 05:45 UTC (permalink / raw)
To: Kyotaro Horiguchi <[email protected]>; +Cc: [email protected]; [email protected]
On Mon, Jun 08, 2026 at 02:20:00PM +0900, Kyotaro Horiguchi wrote:
> At Mon, 4 May 2026 10:23:47 +0800, Chao Li <[email protected]> wrote in
>> I read the code of pg_stat_lock() and played a bit with it. I happened =
>> to notice one thing: the function always returns 12 rows, but the =
>> planner estimates 10 rows:
>
> I'm not convinced this needs to be adjusted.
Neither am I. An estimate does not have to match the exact reality,
especially when it comes to these system functions. Just one example
from pg_proc.dat: pg_stat_get_io() has a prorows of 30, returns 95
rows.
--
Michael
Attachments:
[application/pgp-signature] signature.asc (833B, ../../[email protected]/2-signature.asc)
download
^ permalink raw reply [nested|flat] 48+ messages in thread
* Re: Adjust pg_stat_get_lock() prorows to match lock types
@ 2026-06-08 18:46 Tom Lane <[email protected]>
parent: Kyotaro Horiguchi <[email protected]>
1 sibling, 0 replies; 48+ messages in thread
From: Tom Lane @ 2026-06-08 18:46 UTC (permalink / raw)
To: Kyotaro Horiguchi <[email protected]>; +Cc: [email protected]; [email protected]
Kyotaro Horiguchi <[email protected]> writes:
> This discussion also made me wonder whether the documentation could be
> a bit more explicit that prorows is intended as a planner estimate
> rather than an exact row count.
>> Estimated number of result rows (zero if not proretset)
+1. For the most part, I'd expect that to be obvious from the
fact that it's a constant ... but maybe being explicit would help.
regards, tom lane
^ permalink raw reply [nested|flat] 48+ messages in thread
end of thread, other threads:[~2026-06-08 18:46 UTC | newest]
Thread overview: 48+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2021-02-03 12:00 [PATCH 1/1] Fix a corner-case in COPY FROM backslash processing. Heikki Linnakangas <[email protected]>
2021-02-03 12:00 [PATCH 1/1] Fix a corner-case in COPY FROM backslash processing. Heikki Linnakangas <[email protected]>
2021-02-03 12:00 [PATCH 1/1] Fix a corner-case in COPY FROM backslash processing. Heikki Linnakangas <[email protected]>
2021-02-03 12:00 [PATCH 1/1] Fix a corner-case in COPY FROM backslash processing. Heikki Linnakangas <[email protected]>
2021-02-03 12:00 [PATCH 1/1] Fix a corner-case in COPY FROM backslash processing. Heikki Linnakangas <[email protected]>
2021-02-03 12:00 [PATCH 1/1] Fix a corner-case in COPY FROM backslash processing. Heikki Linnakangas <[email protected]>
2021-02-03 12:00 [PATCH 1/1] Fix a corner-case in COPY FROM backslash processing. Heikki Linnakangas <[email protected]>
2021-02-03 12:00 [PATCH 1/1] Fix a corner-case in COPY FROM backslash processing. Heikki Linnakangas <[email protected]>
2021-02-03 12:00 [PATCH 1/1] Fix a corner-case in COPY FROM backslash processing. Heikki Linnakangas <[email protected]>
2021-02-03 12:00 [PATCH 1/1] Fix a corner-case in COPY FROM backslash processing. Heikki Linnakangas <[email protected]>
2021-02-03 12:00 [PATCH 1/1] Fix a corner-case in COPY FROM backslash processing. Heikki Linnakangas <[email protected]>
2021-02-03 12:00 [PATCH 1/1] Fix a corner-case in COPY FROM backslash processing. Heikki Linnakangas <[email protected]>
2021-02-03 12:00 [PATCH 1/1] Fix a corner-case in COPY FROM backslash processing. Heikki Linnakangas <[email protected]>
2021-02-03 12:00 [PATCH 1/1] Fix a corner-case in COPY FROM backslash processing. Heikki Linnakangas <[email protected]>
2021-02-03 12:00 [PATCH 1/1] Fix a corner-case in COPY FROM backslash processing. Heikki Linnakangas <[email protected]>
2021-02-03 12:00 [PATCH 1/1] Fix a corner-case in COPY FROM backslash processing. Heikki Linnakangas <[email protected]>
2021-02-03 12:00 [PATCH 1/1] Fix a corner-case in COPY FROM backslash processing. Heikki Linnakangas <[email protected]>
2021-02-03 12:00 [PATCH 1/1] Fix a corner-case in COPY FROM backslash processing. Heikki Linnakangas <[email protected]>
2021-02-03 12:00 [PATCH 1/1] Fix a corner-case in COPY FROM backslash processing. Heikki Linnakangas <[email protected]>
2021-02-03 12:00 [PATCH 1/1] Fix a corner-case in COPY FROM backslash processing. Heikki Linnakangas <[email protected]>
2021-02-03 12:00 [PATCH 1/1] Fix a corner-case in COPY FROM backslash processing. Heikki Linnakangas <[email protected]>
2021-02-03 12:00 [PATCH 1/1] Fix a corner-case in COPY FROM backslash processing. Heikki Linnakangas <[email protected]>
2021-02-04 19:36 [PATCH v2 1/1] Fix a corner-case in COPY FROM backslash processing. Heikki Linnakangas <[email protected]>
2021-02-04 19:36 [PATCH v2 1/1] Fix a corner-case in COPY FROM backslash processing. Heikki Linnakangas <[email protected]>
2021-02-04 19:36 [PATCH v2 1/1] Fix a corner-case in COPY FROM backslash processing. Heikki Linnakangas <[email protected]>
2021-02-04 19:36 [PATCH v2 1/1] Fix a corner-case in COPY FROM backslash processing. Heikki Linnakangas <[email protected]>
2021-02-04 19:36 [PATCH v2 1/1] Fix a corner-case in COPY FROM backslash processing. Heikki Linnakangas <[email protected]>
2021-02-04 19:36 [PATCH v2 1/1] Fix a corner-case in COPY FROM backslash processing. Heikki Linnakangas <[email protected]>
2021-02-04 19:36 [PATCH v2 1/1] Fix a corner-case in COPY FROM backslash processing. Heikki Linnakangas <[email protected]>
2021-02-04 19:36 [PATCH v2 1/1] Fix a corner-case in COPY FROM backslash processing. Heikki Linnakangas <[email protected]>
2021-02-04 19:36 [PATCH v2 1/1] Fix a corner-case in COPY FROM backslash processing. Heikki Linnakangas <[email protected]>
2021-02-04 19:36 [PATCH v2 1/1] Fix a corner-case in COPY FROM backslash processing. Heikki Linnakangas <[email protected]>
2021-02-04 19:36 [PATCH v2 1/1] Fix a corner-case in COPY FROM backslash processing. Heikki Linnakangas <[email protected]>
2021-02-04 19:36 [PATCH v2 1/1] Fix a corner-case in COPY FROM backslash processing. Heikki Linnakangas <[email protected]>
2021-02-04 19:36 [PATCH v2 1/1] Fix a corner-case in COPY FROM backslash processing. Heikki Linnakangas <[email protected]>
2021-02-04 19:36 [PATCH v2 1/1] Fix a corner-case in COPY FROM backslash processing. Heikki Linnakangas <[email protected]>
2021-02-04 19:36 [PATCH v2 1/1] Fix a corner-case in COPY FROM backslash processing. Heikki Linnakangas <[email protected]>
2021-02-04 19:36 [PATCH v2 1/1] Fix a corner-case in COPY FROM backslash processing. Heikki Linnakangas <[email protected]>
2021-02-04 19:36 [PATCH v2 1/1] Fix a corner-case in COPY FROM backslash processing. Heikki Linnakangas <[email protected]>
2021-02-04 19:36 [PATCH v2 1/1] Fix a corner-case in COPY FROM backslash processing. Heikki Linnakangas <[email protected]>
2021-02-04 19:36 [PATCH v2 1/1] Fix a corner-case in COPY FROM backslash processing. Heikki Linnakangas <[email protected]>
2021-02-04 19:36 [PATCH v2 1/1] Fix a corner-case in COPY FROM backslash processing. Heikki Linnakangas <[email protected]>
2021-02-04 19:36 [PATCH v2 1/1] Fix a corner-case in COPY FROM backslash processing. Heikki Linnakangas <[email protected]>
2021-02-04 19:36 [PATCH v2 1/1] Fix a corner-case in COPY FROM backslash processing. Heikki Linnakangas <[email protected]>
2026-06-06 03:30 ` Re: Adjust pg_stat_get_lock() prorows to match lock types =?utf-8?B?Wml6aHVhbkxpdSBYLU1BTg==?= <[email protected]>
2026-06-08 05:20 ` Re: Adjust pg_stat_get_lock() prorows to match lock types Kyotaro Horiguchi <[email protected]>
2026-06-08 05:45 ` Re: Adjust pg_stat_get_lock() prorows to match lock types Michael Paquier <[email protected]>
2026-06-08 18:46 ` Re: Adjust pg_stat_get_lock() prorows to match lock types Tom Lane <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox