public inbox for [email protected]help / color / mirror / Atom feed
[PATCH 1/1] Fix a corner-case in COPY FROM backslash processing. 45+ messages / 2 participants [nested] [flat]
* [PATCH 1/1] Fix a corner-case in COPY FROM backslash processing. @ 2021-02-03 12:00 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 45+ messages in thread From: Heikki Linnakangas @ 2021-02-03 12:00 UTC (permalink / raw) If a multi-byte character is escaped with a backslash in TEXT mode input, we didn't always treat the escape correctly. If: - a multi-byte character is escaped with a backslash, and - the second byte of the character is 0x5C, i.e. the ASCII code of a backslash (\), and - the next character is a dot (.), then copyreadline we would incorrectly interpret the sequence as an end-of-copy marker (\.). this can only happen in encodings that can "embed" ascii characters as the second byte. one example of such sequence is '\x5ca45c2e666f6f' in Big5 encoding. If you put that in a file, and load it with COPY FROM, you'd incorrectly get an "end-of-copy marker corrupt" error. --- src/backend/commands/copyfromparse.c | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index 798e18e013..c20ec482db 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -1084,7 +1084,7 @@ CopyReadLineText(CopyFromState cstate) break; } else if (!cstate->opts.csv_mode) - + { /* * If we are here, it means we found a backslash followed by * something other than a period. In non-CSV mode, anything @@ -1095,8 +1095,17 @@ CopyReadLineText(CopyFromState cstate) * backslashes are not special, so we want to process the * character after the backslash just like a normal character, * so we don't increment in those cases. + * + * In principle, we would need to use pg_encoding_mblen() to + * skip over the character in some encodings, like at the end + * of the loop. But if it's a multi-byte character, it cannot + * have any special meaning and skipping isn't necessary for + * correctness. We can fall through to process it normally + * instead. */ - raw_buf_ptr++; + if (!IS_HIGHBIT_SET(c2)) + raw_buf_ptr++; + } } /* -- 2.30.0 --------------97F3138F3612F1A4F9200D93-- ^ permalink raw reply [nested|flat] 45+ messages in thread
* [PATCH 1/1] Fix a corner-case in COPY FROM backslash processing. @ 2021-02-03 12:00 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 45+ messages in thread From: Heikki Linnakangas @ 2021-02-03 12:00 UTC (permalink / raw) If a multi-byte character is escaped with a backslash in TEXT mode input, we didn't always treat the escape correctly. If: - a multi-byte character is escaped with a backslash, and - the second byte of the character is 0x5C, i.e. the ASCII code of a backslash (\), and - the next character is a dot (.), then copyreadline we would incorrectly interpret the sequence as an end-of-copy marker (\.). this can only happen in encodings that can "embed" ascii characters as the second byte. one example of such sequence is '\x5ca45c2e666f6f' in Big5 encoding. If you put that in a file, and load it with COPY FROM, you'd incorrectly get an "end-of-copy marker corrupt" error. --- src/backend/commands/copyfromparse.c | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index 798e18e013..c20ec482db 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -1084,7 +1084,7 @@ CopyReadLineText(CopyFromState cstate) break; } else if (!cstate->opts.csv_mode) - + { /* * If we are here, it means we found a backslash followed by * something other than a period. In non-CSV mode, anything @@ -1095,8 +1095,17 @@ CopyReadLineText(CopyFromState cstate) * backslashes are not special, so we want to process the * character after the backslash just like a normal character, * so we don't increment in those cases. + * + * In principle, we would need to use pg_encoding_mblen() to + * skip over the character in some encodings, like at the end + * of the loop. But if it's a multi-byte character, it cannot + * have any special meaning and skipping isn't necessary for + * correctness. We can fall through to process it normally + * instead. */ - raw_buf_ptr++; + if (!IS_HIGHBIT_SET(c2)) + raw_buf_ptr++; + } } /* -- 2.30.0 --------------97F3138F3612F1A4F9200D93-- ^ permalink raw reply [nested|flat] 45+ messages in thread
* [PATCH 1/1] Fix a corner-case in COPY FROM backslash processing. @ 2021-02-03 12:00 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 45+ messages in thread From: Heikki Linnakangas @ 2021-02-03 12:00 UTC (permalink / raw) If a multi-byte character is escaped with a backslash in TEXT mode input, we didn't always treat the escape correctly. If: - a multi-byte character is escaped with a backslash, and - the second byte of the character is 0x5C, i.e. the ASCII code of a backslash (\), and - the next character is a dot (.), then copyreadline we would incorrectly interpret the sequence as an end-of-copy marker (\.). this can only happen in encodings that can "embed" ascii characters as the second byte. one example of such sequence is '\x5ca45c2e666f6f' in Big5 encoding. If you put that in a file, and load it with COPY FROM, you'd incorrectly get an "end-of-copy marker corrupt" error. --- src/backend/commands/copyfromparse.c | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index 798e18e013..c20ec482db 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -1084,7 +1084,7 @@ CopyReadLineText(CopyFromState cstate) break; } else if (!cstate->opts.csv_mode) - + { /* * If we are here, it means we found a backslash followed by * something other than a period. In non-CSV mode, anything @@ -1095,8 +1095,17 @@ CopyReadLineText(CopyFromState cstate) * backslashes are not special, so we want to process the * character after the backslash just like a normal character, * so we don't increment in those cases. + * + * In principle, we would need to use pg_encoding_mblen() to + * skip over the character in some encodings, like at the end + * of the loop. But if it's a multi-byte character, it cannot + * have any special meaning and skipping isn't necessary for + * correctness. We can fall through to process it normally + * instead. */ - raw_buf_ptr++; + if (!IS_HIGHBIT_SET(c2)) + raw_buf_ptr++; + } } /* -- 2.30.0 --------------97F3138F3612F1A4F9200D93-- ^ permalink raw reply [nested|flat] 45+ messages in thread
* [PATCH 1/1] Fix a corner-case in COPY FROM backslash processing. @ 2021-02-03 12:00 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 45+ messages in thread From: Heikki Linnakangas @ 2021-02-03 12:00 UTC (permalink / raw) If a multi-byte character is escaped with a backslash in TEXT mode input, we didn't always treat the escape correctly. If: - a multi-byte character is escaped with a backslash, and - the second byte of the character is 0x5C, i.e. the ASCII code of a backslash (\), and - the next character is a dot (.), then copyreadline we would incorrectly interpret the sequence as an end-of-copy marker (\.). this can only happen in encodings that can "embed" ascii characters as the second byte. one example of such sequence is '\x5ca45c2e666f6f' in Big5 encoding. If you put that in a file, and load it with COPY FROM, you'd incorrectly get an "end-of-copy marker corrupt" error. --- src/backend/commands/copyfromparse.c | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index 798e18e013..c20ec482db 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -1084,7 +1084,7 @@ CopyReadLineText(CopyFromState cstate) break; } else if (!cstate->opts.csv_mode) - + { /* * If we are here, it means we found a backslash followed by * something other than a period. In non-CSV mode, anything @@ -1095,8 +1095,17 @@ CopyReadLineText(CopyFromState cstate) * backslashes are not special, so we want to process the * character after the backslash just like a normal character, * so we don't increment in those cases. + * + * In principle, we would need to use pg_encoding_mblen() to + * skip over the character in some encodings, like at the end + * of the loop. But if it's a multi-byte character, it cannot + * have any special meaning and skipping isn't necessary for + * correctness. We can fall through to process it normally + * instead. */ - raw_buf_ptr++; + if (!IS_HIGHBIT_SET(c2)) + raw_buf_ptr++; + } } /* -- 2.30.0 --------------97F3138F3612F1A4F9200D93-- ^ permalink raw reply [nested|flat] 45+ messages in thread
* [PATCH 1/1] Fix a corner-case in COPY FROM backslash processing. @ 2021-02-03 12:00 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 45+ messages in thread From: Heikki Linnakangas @ 2021-02-03 12:00 UTC (permalink / raw) If a multi-byte character is escaped with a backslash in TEXT mode input, we didn't always treat the escape correctly. If: - a multi-byte character is escaped with a backslash, and - the second byte of the character is 0x5C, i.e. the ASCII code of a backslash (\), and - the next character is a dot (.), then copyreadline we would incorrectly interpret the sequence as an end-of-copy marker (\.). this can only happen in encodings that can "embed" ascii characters as the second byte. one example of such sequence is '\x5ca45c2e666f6f' in Big5 encoding. If you put that in a file, and load it with COPY FROM, you'd incorrectly get an "end-of-copy marker corrupt" error. --- src/backend/commands/copyfromparse.c | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index 798e18e013..c20ec482db 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -1084,7 +1084,7 @@ CopyReadLineText(CopyFromState cstate) break; } else if (!cstate->opts.csv_mode) - + { /* * If we are here, it means we found a backslash followed by * something other than a period. In non-CSV mode, anything @@ -1095,8 +1095,17 @@ CopyReadLineText(CopyFromState cstate) * backslashes are not special, so we want to process the * character after the backslash just like a normal character, * so we don't increment in those cases. + * + * In principle, we would need to use pg_encoding_mblen() to + * skip over the character in some encodings, like at the end + * of the loop. But if it's a multi-byte character, it cannot + * have any special meaning and skipping isn't necessary for + * correctness. We can fall through to process it normally + * instead. */ - raw_buf_ptr++; + if (!IS_HIGHBIT_SET(c2)) + raw_buf_ptr++; + } } /* -- 2.30.0 --------------97F3138F3612F1A4F9200D93-- ^ permalink raw reply [nested|flat] 45+ messages in thread
* [PATCH 1/1] Fix a corner-case in COPY FROM backslash processing. @ 2021-02-03 12:00 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 45+ messages in thread From: Heikki Linnakangas @ 2021-02-03 12:00 UTC (permalink / raw) If a multi-byte character is escaped with a backslash in TEXT mode input, we didn't always treat the escape correctly. If: - a multi-byte character is escaped with a backslash, and - the second byte of the character is 0x5C, i.e. the ASCII code of a backslash (\), and - the next character is a dot (.), then copyreadline we would incorrectly interpret the sequence as an end-of-copy marker (\.). this can only happen in encodings that can "embed" ascii characters as the second byte. one example of such sequence is '\x5ca45c2e666f6f' in Big5 encoding. If you put that in a file, and load it with COPY FROM, you'd incorrectly get an "end-of-copy marker corrupt" error. --- src/backend/commands/copyfromparse.c | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index 798e18e013..c20ec482db 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -1084,7 +1084,7 @@ CopyReadLineText(CopyFromState cstate) break; } else if (!cstate->opts.csv_mode) - + { /* * If we are here, it means we found a backslash followed by * something other than a period. In non-CSV mode, anything @@ -1095,8 +1095,17 @@ CopyReadLineText(CopyFromState cstate) * backslashes are not special, so we want to process the * character after the backslash just like a normal character, * so we don't increment in those cases. + * + * In principle, we would need to use pg_encoding_mblen() to + * skip over the character in some encodings, like at the end + * of the loop. But if it's a multi-byte character, it cannot + * have any special meaning and skipping isn't necessary for + * correctness. We can fall through to process it normally + * instead. */ - raw_buf_ptr++; + if (!IS_HIGHBIT_SET(c2)) + raw_buf_ptr++; + } } /* -- 2.30.0 --------------97F3138F3612F1A4F9200D93-- ^ permalink raw reply [nested|flat] 45+ messages in thread
* [PATCH 1/1] Fix a corner-case in COPY FROM backslash processing. @ 2021-02-03 12:00 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 45+ messages in thread From: Heikki Linnakangas @ 2021-02-03 12:00 UTC (permalink / raw) If a multi-byte character is escaped with a backslash in TEXT mode input, we didn't always treat the escape correctly. If: - a multi-byte character is escaped with a backslash, and - the second byte of the character is 0x5C, i.e. the ASCII code of a backslash (\), and - the next character is a dot (.), then copyreadline we would incorrectly interpret the sequence as an end-of-copy marker (\.). this can only happen in encodings that can "embed" ascii characters as the second byte. one example of such sequence is '\x5ca45c2e666f6f' in Big5 encoding. If you put that in a file, and load it with COPY FROM, you'd incorrectly get an "end-of-copy marker corrupt" error. --- src/backend/commands/copyfromparse.c | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index 798e18e013..c20ec482db 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -1084,7 +1084,7 @@ CopyReadLineText(CopyFromState cstate) break; } else if (!cstate->opts.csv_mode) - + { /* * If we are here, it means we found a backslash followed by * something other than a period. In non-CSV mode, anything @@ -1095,8 +1095,17 @@ CopyReadLineText(CopyFromState cstate) * backslashes are not special, so we want to process the * character after the backslash just like a normal character, * so we don't increment in those cases. + * + * In principle, we would need to use pg_encoding_mblen() to + * skip over the character in some encodings, like at the end + * of the loop. But if it's a multi-byte character, it cannot + * have any special meaning and skipping isn't necessary for + * correctness. We can fall through to process it normally + * instead. */ - raw_buf_ptr++; + if (!IS_HIGHBIT_SET(c2)) + raw_buf_ptr++; + } } /* -- 2.30.0 --------------97F3138F3612F1A4F9200D93-- ^ permalink raw reply [nested|flat] 45+ messages in thread
* [PATCH 1/1] Fix a corner-case in COPY FROM backslash processing. @ 2021-02-03 12:00 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 45+ messages in thread From: Heikki Linnakangas @ 2021-02-03 12:00 UTC (permalink / raw) If a multi-byte character is escaped with a backslash in TEXT mode input, we didn't always treat the escape correctly. If: - a multi-byte character is escaped with a backslash, and - the second byte of the character is 0x5C, i.e. the ASCII code of a backslash (\), and - the next character is a dot (.), then copyreadline we would incorrectly interpret the sequence as an end-of-copy marker (\.). this can only happen in encodings that can "embed" ascii characters as the second byte. one example of such sequence is '\x5ca45c2e666f6f' in Big5 encoding. If you put that in a file, and load it with COPY FROM, you'd incorrectly get an "end-of-copy marker corrupt" error. --- src/backend/commands/copyfromparse.c | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index 798e18e013..c20ec482db 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -1084,7 +1084,7 @@ CopyReadLineText(CopyFromState cstate) break; } else if (!cstate->opts.csv_mode) - + { /* * If we are here, it means we found a backslash followed by * something other than a period. In non-CSV mode, anything @@ -1095,8 +1095,17 @@ CopyReadLineText(CopyFromState cstate) * backslashes are not special, so we want to process the * character after the backslash just like a normal character, * so we don't increment in those cases. + * + * In principle, we would need to use pg_encoding_mblen() to + * skip over the character in some encodings, like at the end + * of the loop. But if it's a multi-byte character, it cannot + * have any special meaning and skipping isn't necessary for + * correctness. We can fall through to process it normally + * instead. */ - raw_buf_ptr++; + if (!IS_HIGHBIT_SET(c2)) + raw_buf_ptr++; + } } /* -- 2.30.0 --------------97F3138F3612F1A4F9200D93-- ^ permalink raw reply [nested|flat] 45+ messages in thread
* [PATCH 1/1] Fix a corner-case in COPY FROM backslash processing. @ 2021-02-03 12:00 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 45+ messages in thread From: Heikki Linnakangas @ 2021-02-03 12:00 UTC (permalink / raw) If a multi-byte character is escaped with a backslash in TEXT mode input, we didn't always treat the escape correctly. If: - a multi-byte character is escaped with a backslash, and - the second byte of the character is 0x5C, i.e. the ASCII code of a backslash (\), and - the next character is a dot (.), then copyreadline we would incorrectly interpret the sequence as an end-of-copy marker (\.). this can only happen in encodings that can "embed" ascii characters as the second byte. one example of such sequence is '\x5ca45c2e666f6f' in Big5 encoding. If you put that in a file, and load it with COPY FROM, you'd incorrectly get an "end-of-copy marker corrupt" error. --- src/backend/commands/copyfromparse.c | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index 798e18e013..c20ec482db 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -1084,7 +1084,7 @@ CopyReadLineText(CopyFromState cstate) break; } else if (!cstate->opts.csv_mode) - + { /* * If we are here, it means we found a backslash followed by * something other than a period. In non-CSV mode, anything @@ -1095,8 +1095,17 @@ CopyReadLineText(CopyFromState cstate) * backslashes are not special, so we want to process the * character after the backslash just like a normal character, * so we don't increment in those cases. + * + * In principle, we would need to use pg_encoding_mblen() to + * skip over the character in some encodings, like at the end + * of the loop. But if it's a multi-byte character, it cannot + * have any special meaning and skipping isn't necessary for + * correctness. We can fall through to process it normally + * instead. */ - raw_buf_ptr++; + if (!IS_HIGHBIT_SET(c2)) + raw_buf_ptr++; + } } /* -- 2.30.0 --------------97F3138F3612F1A4F9200D93-- ^ permalink raw reply [nested|flat] 45+ messages in thread
* [PATCH 1/1] Fix a corner-case in COPY FROM backslash processing. @ 2021-02-03 12:00 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 45+ messages in thread From: Heikki Linnakangas @ 2021-02-03 12:00 UTC (permalink / raw) If a multi-byte character is escaped with a backslash in TEXT mode input, we didn't always treat the escape correctly. If: - a multi-byte character is escaped with a backslash, and - the second byte of the character is 0x5C, i.e. the ASCII code of a backslash (\), and - the next character is a dot (.), then copyreadline we would incorrectly interpret the sequence as an end-of-copy marker (\.). this can only happen in encodings that can "embed" ascii characters as the second byte. one example of such sequence is '\x5ca45c2e666f6f' in Big5 encoding. If you put that in a file, and load it with COPY FROM, you'd incorrectly get an "end-of-copy marker corrupt" error. --- src/backend/commands/copyfromparse.c | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index 798e18e013..c20ec482db 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -1084,7 +1084,7 @@ CopyReadLineText(CopyFromState cstate) break; } else if (!cstate->opts.csv_mode) - + { /* * If we are here, it means we found a backslash followed by * something other than a period. In non-CSV mode, anything @@ -1095,8 +1095,17 @@ CopyReadLineText(CopyFromState cstate) * backslashes are not special, so we want to process the * character after the backslash just like a normal character, * so we don't increment in those cases. + * + * In principle, we would need to use pg_encoding_mblen() to + * skip over the character in some encodings, like at the end + * of the loop. But if it's a multi-byte character, it cannot + * have any special meaning and skipping isn't necessary for + * correctness. We can fall through to process it normally + * instead. */ - raw_buf_ptr++; + if (!IS_HIGHBIT_SET(c2)) + raw_buf_ptr++; + } } /* -- 2.30.0 --------------97F3138F3612F1A4F9200D93-- ^ permalink raw reply [nested|flat] 45+ messages in thread
* [PATCH 1/1] Fix a corner-case in COPY FROM backslash processing. @ 2021-02-03 12:00 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 45+ messages in thread From: Heikki Linnakangas @ 2021-02-03 12:00 UTC (permalink / raw) If a multi-byte character is escaped with a backslash in TEXT mode input, we didn't always treat the escape correctly. If: - a multi-byte character is escaped with a backslash, and - the second byte of the character is 0x5C, i.e. the ASCII code of a backslash (\), and - the next character is a dot (.), then copyreadline we would incorrectly interpret the sequence as an end-of-copy marker (\.). this can only happen in encodings that can "embed" ascii characters as the second byte. one example of such sequence is '\x5ca45c2e666f6f' in Big5 encoding. If you put that in a file, and load it with COPY FROM, you'd incorrectly get an "end-of-copy marker corrupt" error. --- src/backend/commands/copyfromparse.c | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index 798e18e013..c20ec482db 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -1084,7 +1084,7 @@ CopyReadLineText(CopyFromState cstate) break; } else if (!cstate->opts.csv_mode) - + { /* * If we are here, it means we found a backslash followed by * something other than a period. In non-CSV mode, anything @@ -1095,8 +1095,17 @@ CopyReadLineText(CopyFromState cstate) * backslashes are not special, so we want to process the * character after the backslash just like a normal character, * so we don't increment in those cases. + * + * In principle, we would need to use pg_encoding_mblen() to + * skip over the character in some encodings, like at the end + * of the loop. But if it's a multi-byte character, it cannot + * have any special meaning and skipping isn't necessary for + * correctness. We can fall through to process it normally + * instead. */ - raw_buf_ptr++; + if (!IS_HIGHBIT_SET(c2)) + raw_buf_ptr++; + } } /* -- 2.30.0 --------------97F3138F3612F1A4F9200D93-- ^ permalink raw reply [nested|flat] 45+ messages in thread
* [PATCH 1/1] Fix a corner-case in COPY FROM backslash processing. @ 2021-02-03 12:00 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 45+ messages in thread From: Heikki Linnakangas @ 2021-02-03 12:00 UTC (permalink / raw) If a multi-byte character is escaped with a backslash in TEXT mode input, we didn't always treat the escape correctly. If: - a multi-byte character is escaped with a backslash, and - the second byte of the character is 0x5C, i.e. the ASCII code of a backslash (\), and - the next character is a dot (.), then copyreadline we would incorrectly interpret the sequence as an end-of-copy marker (\.). this can only happen in encodings that can "embed" ascii characters as the second byte. one example of such sequence is '\x5ca45c2e666f6f' in Big5 encoding. If you put that in a file, and load it with COPY FROM, you'd incorrectly get an "end-of-copy marker corrupt" error. --- src/backend/commands/copyfromparse.c | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index 798e18e013..c20ec482db 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -1084,7 +1084,7 @@ CopyReadLineText(CopyFromState cstate) break; } else if (!cstate->opts.csv_mode) - + { /* * If we are here, it means we found a backslash followed by * something other than a period. In non-CSV mode, anything @@ -1095,8 +1095,17 @@ CopyReadLineText(CopyFromState cstate) * backslashes are not special, so we want to process the * character after the backslash just like a normal character, * so we don't increment in those cases. + * + * In principle, we would need to use pg_encoding_mblen() to + * skip over the character in some encodings, like at the end + * of the loop. But if it's a multi-byte character, it cannot + * have any special meaning and skipping isn't necessary for + * correctness. We can fall through to process it normally + * instead. */ - raw_buf_ptr++; + if (!IS_HIGHBIT_SET(c2)) + raw_buf_ptr++; + } } /* -- 2.30.0 --------------97F3138F3612F1A4F9200D93-- ^ permalink raw reply [nested|flat] 45+ messages in thread
* [PATCH 1/1] Fix a corner-case in COPY FROM backslash processing. @ 2021-02-03 12:00 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 45+ messages in thread From: Heikki Linnakangas @ 2021-02-03 12:00 UTC (permalink / raw) If a multi-byte character is escaped with a backslash in TEXT mode input, we didn't always treat the escape correctly. If: - a multi-byte character is escaped with a backslash, and - the second byte of the character is 0x5C, i.e. the ASCII code of a backslash (\), and - the next character is a dot (.), then copyreadline we would incorrectly interpret the sequence as an end-of-copy marker (\.). this can only happen in encodings that can "embed" ascii characters as the second byte. one example of such sequence is '\x5ca45c2e666f6f' in Big5 encoding. If you put that in a file, and load it with COPY FROM, you'd incorrectly get an "end-of-copy marker corrupt" error. --- src/backend/commands/copyfromparse.c | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index 798e18e013..c20ec482db 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -1084,7 +1084,7 @@ CopyReadLineText(CopyFromState cstate) break; } else if (!cstate->opts.csv_mode) - + { /* * If we are here, it means we found a backslash followed by * something other than a period. In non-CSV mode, anything @@ -1095,8 +1095,17 @@ CopyReadLineText(CopyFromState cstate) * backslashes are not special, so we want to process the * character after the backslash just like a normal character, * so we don't increment in those cases. + * + * In principle, we would need to use pg_encoding_mblen() to + * skip over the character in some encodings, like at the end + * of the loop. But if it's a multi-byte character, it cannot + * have any special meaning and skipping isn't necessary for + * correctness. We can fall through to process it normally + * instead. */ - raw_buf_ptr++; + if (!IS_HIGHBIT_SET(c2)) + raw_buf_ptr++; + } } /* -- 2.30.0 --------------97F3138F3612F1A4F9200D93-- ^ permalink raw reply [nested|flat] 45+ messages in thread
* [PATCH 1/1] Fix a corner-case in COPY FROM backslash processing. @ 2021-02-03 12:00 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 45+ messages in thread From: Heikki Linnakangas @ 2021-02-03 12:00 UTC (permalink / raw) If a multi-byte character is escaped with a backslash in TEXT mode input, we didn't always treat the escape correctly. If: - a multi-byte character is escaped with a backslash, and - the second byte of the character is 0x5C, i.e. the ASCII code of a backslash (\), and - the next character is a dot (.), then copyreadline we would incorrectly interpret the sequence as an end-of-copy marker (\.). this can only happen in encodings that can "embed" ascii characters as the second byte. one example of such sequence is '\x5ca45c2e666f6f' in Big5 encoding. If you put that in a file, and load it with COPY FROM, you'd incorrectly get an "end-of-copy marker corrupt" error. --- src/backend/commands/copyfromparse.c | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index 798e18e013..c20ec482db 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -1084,7 +1084,7 @@ CopyReadLineText(CopyFromState cstate) break; } else if (!cstate->opts.csv_mode) - + { /* * If we are here, it means we found a backslash followed by * something other than a period. In non-CSV mode, anything @@ -1095,8 +1095,17 @@ CopyReadLineText(CopyFromState cstate) * backslashes are not special, so we want to process the * character after the backslash just like a normal character, * so we don't increment in those cases. + * + * In principle, we would need to use pg_encoding_mblen() to + * skip over the character in some encodings, like at the end + * of the loop. But if it's a multi-byte character, it cannot + * have any special meaning and skipping isn't necessary for + * correctness. We can fall through to process it normally + * instead. */ - raw_buf_ptr++; + if (!IS_HIGHBIT_SET(c2)) + raw_buf_ptr++; + } } /* -- 2.30.0 --------------97F3138F3612F1A4F9200D93-- ^ permalink raw reply [nested|flat] 45+ messages in thread
* [PATCH 1/1] Fix a corner-case in COPY FROM backslash processing. @ 2021-02-03 12:00 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 45+ messages in thread From: Heikki Linnakangas @ 2021-02-03 12:00 UTC (permalink / raw) If a multi-byte character is escaped with a backslash in TEXT mode input, we didn't always treat the escape correctly. If: - a multi-byte character is escaped with a backslash, and - the second byte of the character is 0x5C, i.e. the ASCII code of a backslash (\), and - the next character is a dot (.), then copyreadline we would incorrectly interpret the sequence as an end-of-copy marker (\.). this can only happen in encodings that can "embed" ascii characters as the second byte. one example of such sequence is '\x5ca45c2e666f6f' in Big5 encoding. If you put that in a file, and load it with COPY FROM, you'd incorrectly get an "end-of-copy marker corrupt" error. --- src/backend/commands/copyfromparse.c | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index 798e18e013..c20ec482db 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -1084,7 +1084,7 @@ CopyReadLineText(CopyFromState cstate) break; } else if (!cstate->opts.csv_mode) - + { /* * If we are here, it means we found a backslash followed by * something other than a period. In non-CSV mode, anything @@ -1095,8 +1095,17 @@ CopyReadLineText(CopyFromState cstate) * backslashes are not special, so we want to process the * character after the backslash just like a normal character, * so we don't increment in those cases. + * + * In principle, we would need to use pg_encoding_mblen() to + * skip over the character in some encodings, like at the end + * of the loop. But if it's a multi-byte character, it cannot + * have any special meaning and skipping isn't necessary for + * correctness. We can fall through to process it normally + * instead. */ - raw_buf_ptr++; + if (!IS_HIGHBIT_SET(c2)) + raw_buf_ptr++; + } } /* -- 2.30.0 --------------97F3138F3612F1A4F9200D93-- ^ permalink raw reply [nested|flat] 45+ messages in thread
* [PATCH 1/1] Fix a corner-case in COPY FROM backslash processing. @ 2021-02-03 12:00 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 45+ messages in thread From: Heikki Linnakangas @ 2021-02-03 12:00 UTC (permalink / raw) If a multi-byte character is escaped with a backslash in TEXT mode input, we didn't always treat the escape correctly. If: - a multi-byte character is escaped with a backslash, and - the second byte of the character is 0x5C, i.e. the ASCII code of a backslash (\), and - the next character is a dot (.), then copyreadline we would incorrectly interpret the sequence as an end-of-copy marker (\.). this can only happen in encodings that can "embed" ascii characters as the second byte. one example of such sequence is '\x5ca45c2e666f6f' in Big5 encoding. If you put that in a file, and load it with COPY FROM, you'd incorrectly get an "end-of-copy marker corrupt" error. --- src/backend/commands/copyfromparse.c | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index 798e18e013..c20ec482db 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -1084,7 +1084,7 @@ CopyReadLineText(CopyFromState cstate) break; } else if (!cstate->opts.csv_mode) - + { /* * If we are here, it means we found a backslash followed by * something other than a period. In non-CSV mode, anything @@ -1095,8 +1095,17 @@ CopyReadLineText(CopyFromState cstate) * backslashes are not special, so we want to process the * character after the backslash just like a normal character, * so we don't increment in those cases. + * + * In principle, we would need to use pg_encoding_mblen() to + * skip over the character in some encodings, like at the end + * of the loop. But if it's a multi-byte character, it cannot + * have any special meaning and skipping isn't necessary for + * correctness. We can fall through to process it normally + * instead. */ - raw_buf_ptr++; + if (!IS_HIGHBIT_SET(c2)) + raw_buf_ptr++; + } } /* -- 2.30.0 --------------97F3138F3612F1A4F9200D93-- ^ permalink raw reply [nested|flat] 45+ messages in thread
* [PATCH 1/1] Fix a corner-case in COPY FROM backslash processing. @ 2021-02-03 12:00 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 45+ messages in thread From: Heikki Linnakangas @ 2021-02-03 12:00 UTC (permalink / raw) If a multi-byte character is escaped with a backslash in TEXT mode input, we didn't always treat the escape correctly. If: - a multi-byte character is escaped with a backslash, and - the second byte of the character is 0x5C, i.e. the ASCII code of a backslash (\), and - the next character is a dot (.), then copyreadline we would incorrectly interpret the sequence as an end-of-copy marker (\.). this can only happen in encodings that can "embed" ascii characters as the second byte. one example of such sequence is '\x5ca45c2e666f6f' in Big5 encoding. If you put that in a file, and load it with COPY FROM, you'd incorrectly get an "end-of-copy marker corrupt" error. --- src/backend/commands/copyfromparse.c | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index 798e18e013..c20ec482db 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -1084,7 +1084,7 @@ CopyReadLineText(CopyFromState cstate) break; } else if (!cstate->opts.csv_mode) - + { /* * If we are here, it means we found a backslash followed by * something other than a period. In non-CSV mode, anything @@ -1095,8 +1095,17 @@ CopyReadLineText(CopyFromState cstate) * backslashes are not special, so we want to process the * character after the backslash just like a normal character, * so we don't increment in those cases. + * + * In principle, we would need to use pg_encoding_mblen() to + * skip over the character in some encodings, like at the end + * of the loop. But if it's a multi-byte character, it cannot + * have any special meaning and skipping isn't necessary for + * correctness. We can fall through to process it normally + * instead. */ - raw_buf_ptr++; + if (!IS_HIGHBIT_SET(c2)) + raw_buf_ptr++; + } } /* -- 2.30.0 --------------97F3138F3612F1A4F9200D93-- ^ permalink raw reply [nested|flat] 45+ messages in thread
* [PATCH 1/1] Fix a corner-case in COPY FROM backslash processing. @ 2021-02-03 12:00 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 45+ messages in thread From: Heikki Linnakangas @ 2021-02-03 12:00 UTC (permalink / raw) If a multi-byte character is escaped with a backslash in TEXT mode input, we didn't always treat the escape correctly. If: - a multi-byte character is escaped with a backslash, and - the second byte of the character is 0x5C, i.e. the ASCII code of a backslash (\), and - the next character is a dot (.), then copyreadline we would incorrectly interpret the sequence as an end-of-copy marker (\.). this can only happen in encodings that can "embed" ascii characters as the second byte. one example of such sequence is '\x5ca45c2e666f6f' in Big5 encoding. If you put that in a file, and load it with COPY FROM, you'd incorrectly get an "end-of-copy marker corrupt" error. --- src/backend/commands/copyfromparse.c | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index 798e18e013..c20ec482db 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -1084,7 +1084,7 @@ CopyReadLineText(CopyFromState cstate) break; } else if (!cstate->opts.csv_mode) - + { /* * If we are here, it means we found a backslash followed by * something other than a period. In non-CSV mode, anything @@ -1095,8 +1095,17 @@ CopyReadLineText(CopyFromState cstate) * backslashes are not special, so we want to process the * character after the backslash just like a normal character, * so we don't increment in those cases. + * + * In principle, we would need to use pg_encoding_mblen() to + * skip over the character in some encodings, like at the end + * of the loop. But if it's a multi-byte character, it cannot + * have any special meaning and skipping isn't necessary for + * correctness. We can fall through to process it normally + * instead. */ - raw_buf_ptr++; + if (!IS_HIGHBIT_SET(c2)) + raw_buf_ptr++; + } } /* -- 2.30.0 --------------97F3138F3612F1A4F9200D93-- ^ permalink raw reply [nested|flat] 45+ messages in thread
* [PATCH 1/1] Fix a corner-case in COPY FROM backslash processing. @ 2021-02-03 12:00 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 45+ messages in thread From: Heikki Linnakangas @ 2021-02-03 12:00 UTC (permalink / raw) If a multi-byte character is escaped with a backslash in TEXT mode input, we didn't always treat the escape correctly. If: - a multi-byte character is escaped with a backslash, and - the second byte of the character is 0x5C, i.e. the ASCII code of a backslash (\), and - the next character is a dot (.), then copyreadline we would incorrectly interpret the sequence as an end-of-copy marker (\.). this can only happen in encodings that can "embed" ascii characters as the second byte. one example of such sequence is '\x5ca45c2e666f6f' in Big5 encoding. If you put that in a file, and load it with COPY FROM, you'd incorrectly get an "end-of-copy marker corrupt" error. --- src/backend/commands/copyfromparse.c | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index 798e18e013..c20ec482db 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -1084,7 +1084,7 @@ CopyReadLineText(CopyFromState cstate) break; } else if (!cstate->opts.csv_mode) - + { /* * If we are here, it means we found a backslash followed by * something other than a period. In non-CSV mode, anything @@ -1095,8 +1095,17 @@ CopyReadLineText(CopyFromState cstate) * backslashes are not special, so we want to process the * character after the backslash just like a normal character, * so we don't increment in those cases. + * + * In principle, we would need to use pg_encoding_mblen() to + * skip over the character in some encodings, like at the end + * of the loop. But if it's a multi-byte character, it cannot + * have any special meaning and skipping isn't necessary for + * correctness. We can fall through to process it normally + * instead. */ - raw_buf_ptr++; + if (!IS_HIGHBIT_SET(c2)) + raw_buf_ptr++; + } } /* -- 2.30.0 --------------97F3138F3612F1A4F9200D93-- ^ permalink raw reply [nested|flat] 45+ messages in thread
* [PATCH 1/1] Fix a corner-case in COPY FROM backslash processing. @ 2021-02-03 12:00 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 45+ messages in thread From: Heikki Linnakangas @ 2021-02-03 12:00 UTC (permalink / raw) If a multi-byte character is escaped with a backslash in TEXT mode input, we didn't always treat the escape correctly. If: - a multi-byte character is escaped with a backslash, and - the second byte of the character is 0x5C, i.e. the ASCII code of a backslash (\), and - the next character is a dot (.), then copyreadline we would incorrectly interpret the sequence as an end-of-copy marker (\.). this can only happen in encodings that can "embed" ascii characters as the second byte. one example of such sequence is '\x5ca45c2e666f6f' in Big5 encoding. If you put that in a file, and load it with COPY FROM, you'd incorrectly get an "end-of-copy marker corrupt" error. --- src/backend/commands/copyfromparse.c | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index 798e18e013..c20ec482db 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -1084,7 +1084,7 @@ CopyReadLineText(CopyFromState cstate) break; } else if (!cstate->opts.csv_mode) - + { /* * If we are here, it means we found a backslash followed by * something other than a period. In non-CSV mode, anything @@ -1095,8 +1095,17 @@ CopyReadLineText(CopyFromState cstate) * backslashes are not special, so we want to process the * character after the backslash just like a normal character, * so we don't increment in those cases. + * + * In principle, we would need to use pg_encoding_mblen() to + * skip over the character in some encodings, like at the end + * of the loop. But if it's a multi-byte character, it cannot + * have any special meaning and skipping isn't necessary for + * correctness. We can fall through to process it normally + * instead. */ - raw_buf_ptr++; + if (!IS_HIGHBIT_SET(c2)) + raw_buf_ptr++; + } } /* -- 2.30.0 --------------97F3138F3612F1A4F9200D93-- ^ permalink raw reply [nested|flat] 45+ messages in thread
* [PATCH 1/1] Fix a corner-case in COPY FROM backslash processing. @ 2021-02-03 12:00 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 45+ messages in thread From: Heikki Linnakangas @ 2021-02-03 12:00 UTC (permalink / raw) If a multi-byte character is escaped with a backslash in TEXT mode input, we didn't always treat the escape correctly. If: - a multi-byte character is escaped with a backslash, and - the second byte of the character is 0x5C, i.e. the ASCII code of a backslash (\), and - the next character is a dot (.), then copyreadline we would incorrectly interpret the sequence as an end-of-copy marker (\.). this can only happen in encodings that can "embed" ascii characters as the second byte. one example of such sequence is '\x5ca45c2e666f6f' in Big5 encoding. If you put that in a file, and load it with COPY FROM, you'd incorrectly get an "end-of-copy marker corrupt" error. --- src/backend/commands/copyfromparse.c | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index 798e18e013..c20ec482db 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -1084,7 +1084,7 @@ CopyReadLineText(CopyFromState cstate) break; } else if (!cstate->opts.csv_mode) - + { /* * If we are here, it means we found a backslash followed by * something other than a period. In non-CSV mode, anything @@ -1095,8 +1095,17 @@ CopyReadLineText(CopyFromState cstate) * backslashes are not special, so we want to process the * character after the backslash just like a normal character, * so we don't increment in those cases. + * + * In principle, we would need to use pg_encoding_mblen() to + * skip over the character in some encodings, like at the end + * of the loop. But if it's a multi-byte character, it cannot + * have any special meaning and skipping isn't necessary for + * correctness. We can fall through to process it normally + * instead. */ - raw_buf_ptr++; + if (!IS_HIGHBIT_SET(c2)) + raw_buf_ptr++; + } } /* -- 2.30.0 --------------97F3138F3612F1A4F9200D93-- ^ permalink raw reply [nested|flat] 45+ messages in thread
* [PATCH 1/1] Fix a corner-case in COPY FROM backslash processing. @ 2021-02-03 12:00 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 45+ messages in thread From: Heikki Linnakangas @ 2021-02-03 12:00 UTC (permalink / raw) If a multi-byte character is escaped with a backslash in TEXT mode input, we didn't always treat the escape correctly. If: - a multi-byte character is escaped with a backslash, and - the second byte of the character is 0x5C, i.e. the ASCII code of a backslash (\), and - the next character is a dot (.), then copyreadline we would incorrectly interpret the sequence as an end-of-copy marker (\.). this can only happen in encodings that can "embed" ascii characters as the second byte. one example of such sequence is '\x5ca45c2e666f6f' in Big5 encoding. If you put that in a file, and load it with COPY FROM, you'd incorrectly get an "end-of-copy marker corrupt" error. --- src/backend/commands/copyfromparse.c | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index 798e18e013..c20ec482db 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -1084,7 +1084,7 @@ CopyReadLineText(CopyFromState cstate) break; } else if (!cstate->opts.csv_mode) - + { /* * If we are here, it means we found a backslash followed by * something other than a period. In non-CSV mode, anything @@ -1095,8 +1095,17 @@ CopyReadLineText(CopyFromState cstate) * backslashes are not special, so we want to process the * character after the backslash just like a normal character, * so we don't increment in those cases. + * + * In principle, we would need to use pg_encoding_mblen() to + * skip over the character in some encodings, like at the end + * of the loop. But if it's a multi-byte character, it cannot + * have any special meaning and skipping isn't necessary for + * correctness. We can fall through to process it normally + * instead. */ - raw_buf_ptr++; + if (!IS_HIGHBIT_SET(c2)) + raw_buf_ptr++; + } } /* -- 2.30.0 --------------97F3138F3612F1A4F9200D93-- ^ permalink raw reply [nested|flat] 45+ messages in thread
* [PATCH v2 1/1] Fix a corner-case in COPY FROM backslash processing. @ 2021-02-04 19:36 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 45+ messages in thread From: Heikki Linnakangas @ 2021-02-04 19:36 UTC (permalink / raw) If a multi-byte character is escaped with a backslash in TEXT mode input, we didn't always treat the escape correctly. If: - a multi-byte character is escaped with a backslash, and - the second byte of the character is 0x5C, i.e. the ASCII code of a backslash (\), and - the next character is a dot (.), then CopyReadLineText function would incorrectly interpret the sequence as an end-of-copy marker (\.). This can only happen in encodings that can "embed" ascii characters as the second byte. One example of such sequence is '\x5ca45c2e666f6f' in Big5 encoding. If you put that in a file, and load it with COPY FROM, you'd incorrectly get an "end-of-copy marker corrupt" error. Backpatch to all supported versions. Reviewed-by: John Naylor, Kyotaro Horiguchi Discussion: https://www.postgresql.org/message-id/a897f84f-8dca-8798-3139-07da5bb38728%40iki.fi --- src/backend/commands/copyfromparse.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index b843d315b1..315b16fd7a 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -1084,7 +1084,7 @@ CopyReadLineText(CopyFromState cstate) break; } else if (!cstate->opts.csv_mode) - + { /* * If we are here, it means we found a backslash followed by * something other than a period. In non-CSV mode, anything @@ -1095,8 +1095,16 @@ CopyReadLineText(CopyFromState cstate) * backslashes are not special, so we want to process the * character after the backslash just like a normal character, * so we don't increment in those cases. + * + * Set 'c' to skip whole character correctly in multi-byte + * encodings. If we don't have the whole character in the + * buffer yet, we might loop back to process it, after all, + * but that's OK because multi-byte characters cannot have any + * special meaning. */ raw_buf_ptr++; + c = c2; + } } /* -- 2.30.0 --------------CFC80B40DFDD66DF067F288D-- ^ permalink raw reply [nested|flat] 45+ messages in thread
* [PATCH v2 1/1] Fix a corner-case in COPY FROM backslash processing. @ 2021-02-04 19:36 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 45+ messages in thread From: Heikki Linnakangas @ 2021-02-04 19:36 UTC (permalink / raw) If a multi-byte character is escaped with a backslash in TEXT mode input, we didn't always treat the escape correctly. If: - a multi-byte character is escaped with a backslash, and - the second byte of the character is 0x5C, i.e. the ASCII code of a backslash (\), and - the next character is a dot (.), then CopyReadLineText function would incorrectly interpret the sequence as an end-of-copy marker (\.). This can only happen in encodings that can "embed" ascii characters as the second byte. One example of such sequence is '\x5ca45c2e666f6f' in Big5 encoding. If you put that in a file, and load it with COPY FROM, you'd incorrectly get an "end-of-copy marker corrupt" error. Backpatch to all supported versions. Reviewed-by: John Naylor, Kyotaro Horiguchi Discussion: https://www.postgresql.org/message-id/a897f84f-8dca-8798-3139-07da5bb38728%40iki.fi --- src/backend/commands/copyfromparse.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index b843d315b1..315b16fd7a 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -1084,7 +1084,7 @@ CopyReadLineText(CopyFromState cstate) break; } else if (!cstate->opts.csv_mode) - + { /* * If we are here, it means we found a backslash followed by * something other than a period. In non-CSV mode, anything @@ -1095,8 +1095,16 @@ CopyReadLineText(CopyFromState cstate) * backslashes are not special, so we want to process the * character after the backslash just like a normal character, * so we don't increment in those cases. + * + * Set 'c' to skip whole character correctly in multi-byte + * encodings. If we don't have the whole character in the + * buffer yet, we might loop back to process it, after all, + * but that's OK because multi-byte characters cannot have any + * special meaning. */ raw_buf_ptr++; + c = c2; + } } /* -- 2.30.0 --------------CFC80B40DFDD66DF067F288D-- ^ permalink raw reply [nested|flat] 45+ messages in thread
* [PATCH v2 1/1] Fix a corner-case in COPY FROM backslash processing. @ 2021-02-04 19:36 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 45+ messages in thread From: Heikki Linnakangas @ 2021-02-04 19:36 UTC (permalink / raw) If a multi-byte character is escaped with a backslash in TEXT mode input, we didn't always treat the escape correctly. If: - a multi-byte character is escaped with a backslash, and - the second byte of the character is 0x5C, i.e. the ASCII code of a backslash (\), and - the next character is a dot (.), then CopyReadLineText function would incorrectly interpret the sequence as an end-of-copy marker (\.). This can only happen in encodings that can "embed" ascii characters as the second byte. One example of such sequence is '\x5ca45c2e666f6f' in Big5 encoding. If you put that in a file, and load it with COPY FROM, you'd incorrectly get an "end-of-copy marker corrupt" error. Backpatch to all supported versions. Reviewed-by: John Naylor, Kyotaro Horiguchi Discussion: https://www.postgresql.org/message-id/a897f84f-8dca-8798-3139-07da5bb38728%40iki.fi --- src/backend/commands/copyfromparse.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index b843d315b1..315b16fd7a 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -1084,7 +1084,7 @@ CopyReadLineText(CopyFromState cstate) break; } else if (!cstate->opts.csv_mode) - + { /* * If we are here, it means we found a backslash followed by * something other than a period. In non-CSV mode, anything @@ -1095,8 +1095,16 @@ CopyReadLineText(CopyFromState cstate) * backslashes are not special, so we want to process the * character after the backslash just like a normal character, * so we don't increment in those cases. + * + * Set 'c' to skip whole character correctly in multi-byte + * encodings. If we don't have the whole character in the + * buffer yet, we might loop back to process it, after all, + * but that's OK because multi-byte characters cannot have any + * special meaning. */ raw_buf_ptr++; + c = c2; + } } /* -- 2.30.0 --------------CFC80B40DFDD66DF067F288D-- ^ permalink raw reply [nested|flat] 45+ messages in thread
* [PATCH v2 1/1] Fix a corner-case in COPY FROM backslash processing. @ 2021-02-04 19:36 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 45+ messages in thread From: Heikki Linnakangas @ 2021-02-04 19:36 UTC (permalink / raw) If a multi-byte character is escaped with a backslash in TEXT mode input, we didn't always treat the escape correctly. If: - a multi-byte character is escaped with a backslash, and - the second byte of the character is 0x5C, i.e. the ASCII code of a backslash (\), and - the next character is a dot (.), then CopyReadLineText function would incorrectly interpret the sequence as an end-of-copy marker (\.). This can only happen in encodings that can "embed" ascii characters as the second byte. One example of such sequence is '\x5ca45c2e666f6f' in Big5 encoding. If you put that in a file, and load it with COPY FROM, you'd incorrectly get an "end-of-copy marker corrupt" error. Backpatch to all supported versions. Reviewed-by: John Naylor, Kyotaro Horiguchi Discussion: https://www.postgresql.org/message-id/a897f84f-8dca-8798-3139-07da5bb38728%40iki.fi --- src/backend/commands/copyfromparse.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index b843d315b1..315b16fd7a 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -1084,7 +1084,7 @@ CopyReadLineText(CopyFromState cstate) break; } else if (!cstate->opts.csv_mode) - + { /* * If we are here, it means we found a backslash followed by * something other than a period. In non-CSV mode, anything @@ -1095,8 +1095,16 @@ CopyReadLineText(CopyFromState cstate) * backslashes are not special, so we want to process the * character after the backslash just like a normal character, * so we don't increment in those cases. + * + * Set 'c' to skip whole character correctly in multi-byte + * encodings. If we don't have the whole character in the + * buffer yet, we might loop back to process it, after all, + * but that's OK because multi-byte characters cannot have any + * special meaning. */ raw_buf_ptr++; + c = c2; + } } /* -- 2.30.0 --------------CFC80B40DFDD66DF067F288D-- ^ permalink raw reply [nested|flat] 45+ messages in thread
* [PATCH v2 1/1] Fix a corner-case in COPY FROM backslash processing. @ 2021-02-04 19:36 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 45+ messages in thread From: Heikki Linnakangas @ 2021-02-04 19:36 UTC (permalink / raw) If a multi-byte character is escaped with a backslash in TEXT mode input, we didn't always treat the escape correctly. If: - a multi-byte character is escaped with a backslash, and - the second byte of the character is 0x5C, i.e. the ASCII code of a backslash (\), and - the next character is a dot (.), then CopyReadLineText function would incorrectly interpret the sequence as an end-of-copy marker (\.). This can only happen in encodings that can "embed" ascii characters as the second byte. One example of such sequence is '\x5ca45c2e666f6f' in Big5 encoding. If you put that in a file, and load it with COPY FROM, you'd incorrectly get an "end-of-copy marker corrupt" error. Backpatch to all supported versions. Reviewed-by: John Naylor, Kyotaro Horiguchi Discussion: https://www.postgresql.org/message-id/a897f84f-8dca-8798-3139-07da5bb38728%40iki.fi --- src/backend/commands/copyfromparse.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index b843d315b1..315b16fd7a 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -1084,7 +1084,7 @@ CopyReadLineText(CopyFromState cstate) break; } else if (!cstate->opts.csv_mode) - + { /* * If we are here, it means we found a backslash followed by * something other than a period. In non-CSV mode, anything @@ -1095,8 +1095,16 @@ CopyReadLineText(CopyFromState cstate) * backslashes are not special, so we want to process the * character after the backslash just like a normal character, * so we don't increment in those cases. + * + * Set 'c' to skip whole character correctly in multi-byte + * encodings. If we don't have the whole character in the + * buffer yet, we might loop back to process it, after all, + * but that's OK because multi-byte characters cannot have any + * special meaning. */ raw_buf_ptr++; + c = c2; + } } /* -- 2.30.0 --------------CFC80B40DFDD66DF067F288D-- ^ permalink raw reply [nested|flat] 45+ messages in thread
* [PATCH v2 1/1] Fix a corner-case in COPY FROM backslash processing. @ 2021-02-04 19:36 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 45+ messages in thread From: Heikki Linnakangas @ 2021-02-04 19:36 UTC (permalink / raw) If a multi-byte character is escaped with a backslash in TEXT mode input, we didn't always treat the escape correctly. If: - a multi-byte character is escaped with a backslash, and - the second byte of the character is 0x5C, i.e. the ASCII code of a backslash (\), and - the next character is a dot (.), then CopyReadLineText function would incorrectly interpret the sequence as an end-of-copy marker (\.). This can only happen in encodings that can "embed" ascii characters as the second byte. One example of such sequence is '\x5ca45c2e666f6f' in Big5 encoding. If you put that in a file, and load it with COPY FROM, you'd incorrectly get an "end-of-copy marker corrupt" error. Backpatch to all supported versions. Reviewed-by: John Naylor, Kyotaro Horiguchi Discussion: https://www.postgresql.org/message-id/a897f84f-8dca-8798-3139-07da5bb38728%40iki.fi --- src/backend/commands/copyfromparse.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index b843d315b1..315b16fd7a 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -1084,7 +1084,7 @@ CopyReadLineText(CopyFromState cstate) break; } else if (!cstate->opts.csv_mode) - + { /* * If we are here, it means we found a backslash followed by * something other than a period. In non-CSV mode, anything @@ -1095,8 +1095,16 @@ CopyReadLineText(CopyFromState cstate) * backslashes are not special, so we want to process the * character after the backslash just like a normal character, * so we don't increment in those cases. + * + * Set 'c' to skip whole character correctly in multi-byte + * encodings. If we don't have the whole character in the + * buffer yet, we might loop back to process it, after all, + * but that's OK because multi-byte characters cannot have any + * special meaning. */ raw_buf_ptr++; + c = c2; + } } /* -- 2.30.0 --------------CFC80B40DFDD66DF067F288D-- ^ permalink raw reply [nested|flat] 45+ messages in thread
* [PATCH v2 1/1] Fix a corner-case in COPY FROM backslash processing. @ 2021-02-04 19:36 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 45+ messages in thread From: Heikki Linnakangas @ 2021-02-04 19:36 UTC (permalink / raw) If a multi-byte character is escaped with a backslash in TEXT mode input, we didn't always treat the escape correctly. If: - a multi-byte character is escaped with a backslash, and - the second byte of the character is 0x5C, i.e. the ASCII code of a backslash (\), and - the next character is a dot (.), then CopyReadLineText function would incorrectly interpret the sequence as an end-of-copy marker (\.). This can only happen in encodings that can "embed" ascii characters as the second byte. One example of such sequence is '\x5ca45c2e666f6f' in Big5 encoding. If you put that in a file, and load it with COPY FROM, you'd incorrectly get an "end-of-copy marker corrupt" error. Backpatch to all supported versions. Reviewed-by: John Naylor, Kyotaro Horiguchi Discussion: https://www.postgresql.org/message-id/a897f84f-8dca-8798-3139-07da5bb38728%40iki.fi --- src/backend/commands/copyfromparse.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index b843d315b1..315b16fd7a 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -1084,7 +1084,7 @@ CopyReadLineText(CopyFromState cstate) break; } else if (!cstate->opts.csv_mode) - + { /* * If we are here, it means we found a backslash followed by * something other than a period. In non-CSV mode, anything @@ -1095,8 +1095,16 @@ CopyReadLineText(CopyFromState cstate) * backslashes are not special, so we want to process the * character after the backslash just like a normal character, * so we don't increment in those cases. + * + * Set 'c' to skip whole character correctly in multi-byte + * encodings. If we don't have the whole character in the + * buffer yet, we might loop back to process it, after all, + * but that's OK because multi-byte characters cannot have any + * special meaning. */ raw_buf_ptr++; + c = c2; + } } /* -- 2.30.0 --------------CFC80B40DFDD66DF067F288D-- ^ permalink raw reply [nested|flat] 45+ messages in thread
* [PATCH v2 1/1] Fix a corner-case in COPY FROM backslash processing. @ 2021-02-04 19:36 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 45+ messages in thread From: Heikki Linnakangas @ 2021-02-04 19:36 UTC (permalink / raw) If a multi-byte character is escaped with a backslash in TEXT mode input, we didn't always treat the escape correctly. If: - a multi-byte character is escaped with a backslash, and - the second byte of the character is 0x5C, i.e. the ASCII code of a backslash (\), and - the next character is a dot (.), then CopyReadLineText function would incorrectly interpret the sequence as an end-of-copy marker (\.). This can only happen in encodings that can "embed" ascii characters as the second byte. One example of such sequence is '\x5ca45c2e666f6f' in Big5 encoding. If you put that in a file, and load it with COPY FROM, you'd incorrectly get an "end-of-copy marker corrupt" error. Backpatch to all supported versions. Reviewed-by: John Naylor, Kyotaro Horiguchi Discussion: https://www.postgresql.org/message-id/a897f84f-8dca-8798-3139-07da5bb38728%40iki.fi --- src/backend/commands/copyfromparse.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index b843d315b1..315b16fd7a 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -1084,7 +1084,7 @@ CopyReadLineText(CopyFromState cstate) break; } else if (!cstate->opts.csv_mode) - + { /* * If we are here, it means we found a backslash followed by * something other than a period. In non-CSV mode, anything @@ -1095,8 +1095,16 @@ CopyReadLineText(CopyFromState cstate) * backslashes are not special, so we want to process the * character after the backslash just like a normal character, * so we don't increment in those cases. + * + * Set 'c' to skip whole character correctly in multi-byte + * encodings. If we don't have the whole character in the + * buffer yet, we might loop back to process it, after all, + * but that's OK because multi-byte characters cannot have any + * special meaning. */ raw_buf_ptr++; + c = c2; + } } /* -- 2.30.0 --------------CFC80B40DFDD66DF067F288D-- ^ permalink raw reply [nested|flat] 45+ messages in thread
* [PATCH v2 1/1] Fix a corner-case in COPY FROM backslash processing. @ 2021-02-04 19:36 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 45+ messages in thread From: Heikki Linnakangas @ 2021-02-04 19:36 UTC (permalink / raw) If a multi-byte character is escaped with a backslash in TEXT mode input, we didn't always treat the escape correctly. If: - a multi-byte character is escaped with a backslash, and - the second byte of the character is 0x5C, i.e. the ASCII code of a backslash (\), and - the next character is a dot (.), then CopyReadLineText function would incorrectly interpret the sequence as an end-of-copy marker (\.). This can only happen in encodings that can "embed" ascii characters as the second byte. One example of such sequence is '\x5ca45c2e666f6f' in Big5 encoding. If you put that in a file, and load it with COPY FROM, you'd incorrectly get an "end-of-copy marker corrupt" error. Backpatch to all supported versions. Reviewed-by: John Naylor, Kyotaro Horiguchi Discussion: https://www.postgresql.org/message-id/a897f84f-8dca-8798-3139-07da5bb38728%40iki.fi --- src/backend/commands/copyfromparse.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index b843d315b1..315b16fd7a 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -1084,7 +1084,7 @@ CopyReadLineText(CopyFromState cstate) break; } else if (!cstate->opts.csv_mode) - + { /* * If we are here, it means we found a backslash followed by * something other than a period. In non-CSV mode, anything @@ -1095,8 +1095,16 @@ CopyReadLineText(CopyFromState cstate) * backslashes are not special, so we want to process the * character after the backslash just like a normal character, * so we don't increment in those cases. + * + * Set 'c' to skip whole character correctly in multi-byte + * encodings. If we don't have the whole character in the + * buffer yet, we might loop back to process it, after all, + * but that's OK because multi-byte characters cannot have any + * special meaning. */ raw_buf_ptr++; + c = c2; + } } /* -- 2.30.0 --------------CFC80B40DFDD66DF067F288D-- ^ permalink raw reply [nested|flat] 45+ messages in thread
* [PATCH v2 1/1] Fix a corner-case in COPY FROM backslash processing. @ 2021-02-04 19:36 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 45+ messages in thread From: Heikki Linnakangas @ 2021-02-04 19:36 UTC (permalink / raw) If a multi-byte character is escaped with a backslash in TEXT mode input, we didn't always treat the escape correctly. If: - a multi-byte character is escaped with a backslash, and - the second byte of the character is 0x5C, i.e. the ASCII code of a backslash (\), and - the next character is a dot (.), then CopyReadLineText function would incorrectly interpret the sequence as an end-of-copy marker (\.). This can only happen in encodings that can "embed" ascii characters as the second byte. One example of such sequence is '\x5ca45c2e666f6f' in Big5 encoding. If you put that in a file, and load it with COPY FROM, you'd incorrectly get an "end-of-copy marker corrupt" error. Backpatch to all supported versions. Reviewed-by: John Naylor, Kyotaro Horiguchi Discussion: https://www.postgresql.org/message-id/a897f84f-8dca-8798-3139-07da5bb38728%40iki.fi --- src/backend/commands/copyfromparse.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index b843d315b1..315b16fd7a 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -1084,7 +1084,7 @@ CopyReadLineText(CopyFromState cstate) break; } else if (!cstate->opts.csv_mode) - + { /* * If we are here, it means we found a backslash followed by * something other than a period. In non-CSV mode, anything @@ -1095,8 +1095,16 @@ CopyReadLineText(CopyFromState cstate) * backslashes are not special, so we want to process the * character after the backslash just like a normal character, * so we don't increment in those cases. + * + * Set 'c' to skip whole character correctly in multi-byte + * encodings. If we don't have the whole character in the + * buffer yet, we might loop back to process it, after all, + * but that's OK because multi-byte characters cannot have any + * special meaning. */ raw_buf_ptr++; + c = c2; + } } /* -- 2.30.0 --------------CFC80B40DFDD66DF067F288D-- ^ permalink raw reply [nested|flat] 45+ messages in thread
* [PATCH v2 1/1] Fix a corner-case in COPY FROM backslash processing. @ 2021-02-04 19:36 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 45+ messages in thread From: Heikki Linnakangas @ 2021-02-04 19:36 UTC (permalink / raw) If a multi-byte character is escaped with a backslash in TEXT mode input, we didn't always treat the escape correctly. If: - a multi-byte character is escaped with a backslash, and - the second byte of the character is 0x5C, i.e. the ASCII code of a backslash (\), and - the next character is a dot (.), then CopyReadLineText function would incorrectly interpret the sequence as an end-of-copy marker (\.). This can only happen in encodings that can "embed" ascii characters as the second byte. One example of such sequence is '\x5ca45c2e666f6f' in Big5 encoding. If you put that in a file, and load it with COPY FROM, you'd incorrectly get an "end-of-copy marker corrupt" error. Backpatch to all supported versions. Reviewed-by: John Naylor, Kyotaro Horiguchi Discussion: https://www.postgresql.org/message-id/a897f84f-8dca-8798-3139-07da5bb38728%40iki.fi --- src/backend/commands/copyfromparse.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index b843d315b1..315b16fd7a 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -1084,7 +1084,7 @@ CopyReadLineText(CopyFromState cstate) break; } else if (!cstate->opts.csv_mode) - + { /* * If we are here, it means we found a backslash followed by * something other than a period. In non-CSV mode, anything @@ -1095,8 +1095,16 @@ CopyReadLineText(CopyFromState cstate) * backslashes are not special, so we want to process the * character after the backslash just like a normal character, * so we don't increment in those cases. + * + * Set 'c' to skip whole character correctly in multi-byte + * encodings. If we don't have the whole character in the + * buffer yet, we might loop back to process it, after all, + * but that's OK because multi-byte characters cannot have any + * special meaning. */ raw_buf_ptr++; + c = c2; + } } /* -- 2.30.0 --------------CFC80B40DFDD66DF067F288D-- ^ permalink raw reply [nested|flat] 45+ messages in thread
* [PATCH v2 1/1] Fix a corner-case in COPY FROM backslash processing. @ 2021-02-04 19:36 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 45+ messages in thread From: Heikki Linnakangas @ 2021-02-04 19:36 UTC (permalink / raw) If a multi-byte character is escaped with a backslash in TEXT mode input, we didn't always treat the escape correctly. If: - a multi-byte character is escaped with a backslash, and - the second byte of the character is 0x5C, i.e. the ASCII code of a backslash (\), and - the next character is a dot (.), then CopyReadLineText function would incorrectly interpret the sequence as an end-of-copy marker (\.). This can only happen in encodings that can "embed" ascii characters as the second byte. One example of such sequence is '\x5ca45c2e666f6f' in Big5 encoding. If you put that in a file, and load it with COPY FROM, you'd incorrectly get an "end-of-copy marker corrupt" error. Backpatch to all supported versions. Reviewed-by: John Naylor, Kyotaro Horiguchi Discussion: https://www.postgresql.org/message-id/a897f84f-8dca-8798-3139-07da5bb38728%40iki.fi --- src/backend/commands/copyfromparse.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index b843d315b1..315b16fd7a 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -1084,7 +1084,7 @@ CopyReadLineText(CopyFromState cstate) break; } else if (!cstate->opts.csv_mode) - + { /* * If we are here, it means we found a backslash followed by * something other than a period. In non-CSV mode, anything @@ -1095,8 +1095,16 @@ CopyReadLineText(CopyFromState cstate) * backslashes are not special, so we want to process the * character after the backslash just like a normal character, * so we don't increment in those cases. + * + * Set 'c' to skip whole character correctly in multi-byte + * encodings. If we don't have the whole character in the + * buffer yet, we might loop back to process it, after all, + * but that's OK because multi-byte characters cannot have any + * special meaning. */ raw_buf_ptr++; + c = c2; + } } /* -- 2.30.0 --------------CFC80B40DFDD66DF067F288D-- ^ permalink raw reply [nested|flat] 45+ messages in thread
* [PATCH v2 1/1] Fix a corner-case in COPY FROM backslash processing. @ 2021-02-04 19:36 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 45+ messages in thread From: Heikki Linnakangas @ 2021-02-04 19:36 UTC (permalink / raw) If a multi-byte character is escaped with a backslash in TEXT mode input, we didn't always treat the escape correctly. If: - a multi-byte character is escaped with a backslash, and - the second byte of the character is 0x5C, i.e. the ASCII code of a backslash (\), and - the next character is a dot (.), then CopyReadLineText function would incorrectly interpret the sequence as an end-of-copy marker (\.). This can only happen in encodings that can "embed" ascii characters as the second byte. One example of such sequence is '\x5ca45c2e666f6f' in Big5 encoding. If you put that in a file, and load it with COPY FROM, you'd incorrectly get an "end-of-copy marker corrupt" error. Backpatch to all supported versions. Reviewed-by: John Naylor, Kyotaro Horiguchi Discussion: https://www.postgresql.org/message-id/a897f84f-8dca-8798-3139-07da5bb38728%40iki.fi --- src/backend/commands/copyfromparse.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index b843d315b1..315b16fd7a 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -1084,7 +1084,7 @@ CopyReadLineText(CopyFromState cstate) break; } else if (!cstate->opts.csv_mode) - + { /* * If we are here, it means we found a backslash followed by * something other than a period. In non-CSV mode, anything @@ -1095,8 +1095,16 @@ CopyReadLineText(CopyFromState cstate) * backslashes are not special, so we want to process the * character after the backslash just like a normal character, * so we don't increment in those cases. + * + * Set 'c' to skip whole character correctly in multi-byte + * encodings. If we don't have the whole character in the + * buffer yet, we might loop back to process it, after all, + * but that's OK because multi-byte characters cannot have any + * special meaning. */ raw_buf_ptr++; + c = c2; + } } /* -- 2.30.0 --------------CFC80B40DFDD66DF067F288D-- ^ permalink raw reply [nested|flat] 45+ messages in thread
* [PATCH v2 1/1] Fix a corner-case in COPY FROM backslash processing. @ 2021-02-04 19:36 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 45+ messages in thread From: Heikki Linnakangas @ 2021-02-04 19:36 UTC (permalink / raw) If a multi-byte character is escaped with a backslash in TEXT mode input, we didn't always treat the escape correctly. If: - a multi-byte character is escaped with a backslash, and - the second byte of the character is 0x5C, i.e. the ASCII code of a backslash (\), and - the next character is a dot (.), then CopyReadLineText function would incorrectly interpret the sequence as an end-of-copy marker (\.). This can only happen in encodings that can "embed" ascii characters as the second byte. One example of such sequence is '\x5ca45c2e666f6f' in Big5 encoding. If you put that in a file, and load it with COPY FROM, you'd incorrectly get an "end-of-copy marker corrupt" error. Backpatch to all supported versions. Reviewed-by: John Naylor, Kyotaro Horiguchi Discussion: https://www.postgresql.org/message-id/a897f84f-8dca-8798-3139-07da5bb38728%40iki.fi --- src/backend/commands/copyfromparse.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index b843d315b1..315b16fd7a 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -1084,7 +1084,7 @@ CopyReadLineText(CopyFromState cstate) break; } else if (!cstate->opts.csv_mode) - + { /* * If we are here, it means we found a backslash followed by * something other than a period. In non-CSV mode, anything @@ -1095,8 +1095,16 @@ CopyReadLineText(CopyFromState cstate) * backslashes are not special, so we want to process the * character after the backslash just like a normal character, * so we don't increment in those cases. + * + * Set 'c' to skip whole character correctly in multi-byte + * encodings. If we don't have the whole character in the + * buffer yet, we might loop back to process it, after all, + * but that's OK because multi-byte characters cannot have any + * special meaning. */ raw_buf_ptr++; + c = c2; + } } /* -- 2.30.0 --------------CFC80B40DFDD66DF067F288D-- ^ permalink raw reply [nested|flat] 45+ messages in thread
* [PATCH v2 1/1] Fix a corner-case in COPY FROM backslash processing. @ 2021-02-04 19:36 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 45+ messages in thread From: Heikki Linnakangas @ 2021-02-04 19:36 UTC (permalink / raw) If a multi-byte character is escaped with a backslash in TEXT mode input, we didn't always treat the escape correctly. If: - a multi-byte character is escaped with a backslash, and - the second byte of the character is 0x5C, i.e. the ASCII code of a backslash (\), and - the next character is a dot (.), then CopyReadLineText function would incorrectly interpret the sequence as an end-of-copy marker (\.). This can only happen in encodings that can "embed" ascii characters as the second byte. One example of such sequence is '\x5ca45c2e666f6f' in Big5 encoding. If you put that in a file, and load it with COPY FROM, you'd incorrectly get an "end-of-copy marker corrupt" error. Backpatch to all supported versions. Reviewed-by: John Naylor, Kyotaro Horiguchi Discussion: https://www.postgresql.org/message-id/a897f84f-8dca-8798-3139-07da5bb38728%40iki.fi --- src/backend/commands/copyfromparse.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index b843d315b1..315b16fd7a 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -1084,7 +1084,7 @@ CopyReadLineText(CopyFromState cstate) break; } else if (!cstate->opts.csv_mode) - + { /* * If we are here, it means we found a backslash followed by * something other than a period. In non-CSV mode, anything @@ -1095,8 +1095,16 @@ CopyReadLineText(CopyFromState cstate) * backslashes are not special, so we want to process the * character after the backslash just like a normal character, * so we don't increment in those cases. + * + * Set 'c' to skip whole character correctly in multi-byte + * encodings. If we don't have the whole character in the + * buffer yet, we might loop back to process it, after all, + * but that's OK because multi-byte characters cannot have any + * special meaning. */ raw_buf_ptr++; + c = c2; + } } /* -- 2.30.0 --------------CFC80B40DFDD66DF067F288D-- ^ permalink raw reply [nested|flat] 45+ messages in thread
* [PATCH v2 1/1] Fix a corner-case in COPY FROM backslash processing. @ 2021-02-04 19:36 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 45+ messages in thread From: Heikki Linnakangas @ 2021-02-04 19:36 UTC (permalink / raw) If a multi-byte character is escaped with a backslash in TEXT mode input, we didn't always treat the escape correctly. If: - a multi-byte character is escaped with a backslash, and - the second byte of the character is 0x5C, i.e. the ASCII code of a backslash (\), and - the next character is a dot (.), then CopyReadLineText function would incorrectly interpret the sequence as an end-of-copy marker (\.). This can only happen in encodings that can "embed" ascii characters as the second byte. One example of such sequence is '\x5ca45c2e666f6f' in Big5 encoding. If you put that in a file, and load it with COPY FROM, you'd incorrectly get an "end-of-copy marker corrupt" error. Backpatch to all supported versions. Reviewed-by: John Naylor, Kyotaro Horiguchi Discussion: https://www.postgresql.org/message-id/a897f84f-8dca-8798-3139-07da5bb38728%40iki.fi --- src/backend/commands/copyfromparse.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index b843d315b1..315b16fd7a 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -1084,7 +1084,7 @@ CopyReadLineText(CopyFromState cstate) break; } else if (!cstate->opts.csv_mode) - + { /* * If we are here, it means we found a backslash followed by * something other than a period. In non-CSV mode, anything @@ -1095,8 +1095,16 @@ CopyReadLineText(CopyFromState cstate) * backslashes are not special, so we want to process the * character after the backslash just like a normal character, * so we don't increment in those cases. + * + * Set 'c' to skip whole character correctly in multi-byte + * encodings. If we don't have the whole character in the + * buffer yet, we might loop back to process it, after all, + * but that's OK because multi-byte characters cannot have any + * special meaning. */ raw_buf_ptr++; + c = c2; + } } /* -- 2.30.0 --------------CFC80B40DFDD66DF067F288D-- ^ permalink raw reply [nested|flat] 45+ messages in thread
* [PATCH v2 1/1] Fix a corner-case in COPY FROM backslash processing. @ 2021-02-04 19:36 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 45+ messages in thread From: Heikki Linnakangas @ 2021-02-04 19:36 UTC (permalink / raw) If a multi-byte character is escaped with a backslash in TEXT mode input, we didn't always treat the escape correctly. If: - a multi-byte character is escaped with a backslash, and - the second byte of the character is 0x5C, i.e. the ASCII code of a backslash (\), and - the next character is a dot (.), then CopyReadLineText function would incorrectly interpret the sequence as an end-of-copy marker (\.). This can only happen in encodings that can "embed" ascii characters as the second byte. One example of such sequence is '\x5ca45c2e666f6f' in Big5 encoding. If you put that in a file, and load it with COPY FROM, you'd incorrectly get an "end-of-copy marker corrupt" error. Backpatch to all supported versions. Reviewed-by: John Naylor, Kyotaro Horiguchi Discussion: https://www.postgresql.org/message-id/a897f84f-8dca-8798-3139-07da5bb38728%40iki.fi --- src/backend/commands/copyfromparse.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index b843d315b1..315b16fd7a 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -1084,7 +1084,7 @@ CopyReadLineText(CopyFromState cstate) break; } else if (!cstate->opts.csv_mode) - + { /* * If we are here, it means we found a backslash followed by * something other than a period. In non-CSV mode, anything @@ -1095,8 +1095,16 @@ CopyReadLineText(CopyFromState cstate) * backslashes are not special, so we want to process the * character after the backslash just like a normal character, * so we don't increment in those cases. + * + * Set 'c' to skip whole character correctly in multi-byte + * encodings. If we don't have the whole character in the + * buffer yet, we might loop back to process it, after all, + * but that's OK because multi-byte characters cannot have any + * special meaning. */ raw_buf_ptr++; + c = c2; + } } /* -- 2.30.0 --------------CFC80B40DFDD66DF067F288D-- ^ permalink raw reply [nested|flat] 45+ messages in thread
* [PATCH v2 1/1] Fix a corner-case in COPY FROM backslash processing. @ 2021-02-04 19:36 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 45+ messages in thread From: Heikki Linnakangas @ 2021-02-04 19:36 UTC (permalink / raw) If a multi-byte character is escaped with a backslash in TEXT mode input, we didn't always treat the escape correctly. If: - a multi-byte character is escaped with a backslash, and - the second byte of the character is 0x5C, i.e. the ASCII code of a backslash (\), and - the next character is a dot (.), then CopyReadLineText function would incorrectly interpret the sequence as an end-of-copy marker (\.). This can only happen in encodings that can "embed" ascii characters as the second byte. One example of such sequence is '\x5ca45c2e666f6f' in Big5 encoding. If you put that in a file, and load it with COPY FROM, you'd incorrectly get an "end-of-copy marker corrupt" error. Backpatch to all supported versions. Reviewed-by: John Naylor, Kyotaro Horiguchi Discussion: https://www.postgresql.org/message-id/a897f84f-8dca-8798-3139-07da5bb38728%40iki.fi --- src/backend/commands/copyfromparse.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index b843d315b1..315b16fd7a 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -1084,7 +1084,7 @@ CopyReadLineText(CopyFromState cstate) break; } else if (!cstate->opts.csv_mode) - + { /* * If we are here, it means we found a backslash followed by * something other than a period. In non-CSV mode, anything @@ -1095,8 +1095,16 @@ CopyReadLineText(CopyFromState cstate) * backslashes are not special, so we want to process the * character after the backslash just like a normal character, * so we don't increment in those cases. + * + * Set 'c' to skip whole character correctly in multi-byte + * encodings. If we don't have the whole character in the + * buffer yet, we might loop back to process it, after all, + * but that's OK because multi-byte characters cannot have any + * special meaning. */ raw_buf_ptr++; + c = c2; + } } /* -- 2.30.0 --------------CFC80B40DFDD66DF067F288D-- ^ permalink raw reply [nested|flat] 45+ messages in thread
* [PATCH v2 1/1] Fix a corner-case in COPY FROM backslash processing. @ 2021-02-04 19:36 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 45+ messages in thread From: Heikki Linnakangas @ 2021-02-04 19:36 UTC (permalink / raw) If a multi-byte character is escaped with a backslash in TEXT mode input, we didn't always treat the escape correctly. If: - a multi-byte character is escaped with a backslash, and - the second byte of the character is 0x5C, i.e. the ASCII code of a backslash (\), and - the next character is a dot (.), then CopyReadLineText function would incorrectly interpret the sequence as an end-of-copy marker (\.). This can only happen in encodings that can "embed" ascii characters as the second byte. One example of such sequence is '\x5ca45c2e666f6f' in Big5 encoding. If you put that in a file, and load it with COPY FROM, you'd incorrectly get an "end-of-copy marker corrupt" error. Backpatch to all supported versions. Reviewed-by: John Naylor, Kyotaro Horiguchi Discussion: https://www.postgresql.org/message-id/a897f84f-8dca-8798-3139-07da5bb38728%40iki.fi --- src/backend/commands/copyfromparse.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index b843d315b1..315b16fd7a 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -1084,7 +1084,7 @@ CopyReadLineText(CopyFromState cstate) break; } else if (!cstate->opts.csv_mode) - + { /* * If we are here, it means we found a backslash followed by * something other than a period. In non-CSV mode, anything @@ -1095,8 +1095,16 @@ CopyReadLineText(CopyFromState cstate) * backslashes are not special, so we want to process the * character after the backslash just like a normal character, * so we don't increment in those cases. + * + * Set 'c' to skip whole character correctly in multi-byte + * encodings. If we don't have the whole character in the + * buffer yet, we might loop back to process it, after all, + * but that's OK because multi-byte characters cannot have any + * special meaning. */ raw_buf_ptr++; + c = c2; + } } /* -- 2.30.0 --------------CFC80B40DFDD66DF067F288D-- ^ permalink raw reply [nested|flat] 45+ messages in thread
* [PATCH v2 1/1] Fix a corner-case in COPY FROM backslash processing. @ 2021-02-04 19:36 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 45+ messages in thread From: Heikki Linnakangas @ 2021-02-04 19:36 UTC (permalink / raw) If a multi-byte character is escaped with a backslash in TEXT mode input, we didn't always treat the escape correctly. If: - a multi-byte character is escaped with a backslash, and - the second byte of the character is 0x5C, i.e. the ASCII code of a backslash (\), and - the next character is a dot (.), then CopyReadLineText function would incorrectly interpret the sequence as an end-of-copy marker (\.). This can only happen in encodings that can "embed" ascii characters as the second byte. One example of such sequence is '\x5ca45c2e666f6f' in Big5 encoding. If you put that in a file, and load it with COPY FROM, you'd incorrectly get an "end-of-copy marker corrupt" error. Backpatch to all supported versions. Reviewed-by: John Naylor, Kyotaro Horiguchi Discussion: https://www.postgresql.org/message-id/a897f84f-8dca-8798-3139-07da5bb38728%40iki.fi --- src/backend/commands/copyfromparse.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index b843d315b1..315b16fd7a 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -1084,7 +1084,7 @@ CopyReadLineText(CopyFromState cstate) break; } else if (!cstate->opts.csv_mode) - + { /* * If we are here, it means we found a backslash followed by * something other than a period. In non-CSV mode, anything @@ -1095,8 +1095,16 @@ CopyReadLineText(CopyFromState cstate) * backslashes are not special, so we want to process the * character after the backslash just like a normal character, * so we don't increment in those cases. + * + * Set 'c' to skip whole character correctly in multi-byte + * encodings. If we don't have the whole character in the + * buffer yet, we might loop back to process it, after all, + * but that's OK because multi-byte characters cannot have any + * special meaning. */ raw_buf_ptr++; + c = c2; + } } /* -- 2.30.0 --------------CFC80B40DFDD66DF067F288D-- ^ permalink raw reply [nested|flat] 45+ messages in thread
* [PATCH v2 1/1] Fix a corner-case in COPY FROM backslash processing. @ 2021-02-04 19:36 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 45+ messages in thread From: Heikki Linnakangas @ 2021-02-04 19:36 UTC (permalink / raw) If a multi-byte character is escaped with a backslash in TEXT mode input, we didn't always treat the escape correctly. If: - a multi-byte character is escaped with a backslash, and - the second byte of the character is 0x5C, i.e. the ASCII code of a backslash (\), and - the next character is a dot (.), then CopyReadLineText function would incorrectly interpret the sequence as an end-of-copy marker (\.). This can only happen in encodings that can "embed" ascii characters as the second byte. One example of such sequence is '\x5ca45c2e666f6f' in Big5 encoding. If you put that in a file, and load it with COPY FROM, you'd incorrectly get an "end-of-copy marker corrupt" error. Backpatch to all supported versions. Reviewed-by: John Naylor, Kyotaro Horiguchi Discussion: https://www.postgresql.org/message-id/a897f84f-8dca-8798-3139-07da5bb38728%40iki.fi --- src/backend/commands/copyfromparse.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index b843d315b1..315b16fd7a 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -1084,7 +1084,7 @@ CopyReadLineText(CopyFromState cstate) break; } else if (!cstate->opts.csv_mode) - + { /* * If we are here, it means we found a backslash followed by * something other than a period. In non-CSV mode, anything @@ -1095,8 +1095,16 @@ CopyReadLineText(CopyFromState cstate) * backslashes are not special, so we want to process the * character after the backslash just like a normal character, * so we don't increment in those cases. + * + * Set 'c' to skip whole character correctly in multi-byte + * encodings. If we don't have the whole character in the + * buffer yet, we might loop back to process it, after all, + * but that's OK because multi-byte characters cannot have any + * special meaning. */ raw_buf_ptr++; + c = c2; + } } /* -- 2.30.0 --------------CFC80B40DFDD66DF067F288D-- ^ permalink raw reply [nested|flat] 45+ messages in thread
* [PATCH v2 1/1] Fix a corner-case in COPY FROM backslash processing. @ 2021-02-04 19:36 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 45+ messages in thread From: Heikki Linnakangas @ 2021-02-04 19:36 UTC (permalink / raw) If a multi-byte character is escaped with a backslash in TEXT mode input, we didn't always treat the escape correctly. If: - a multi-byte character is escaped with a backslash, and - the second byte of the character is 0x5C, i.e. the ASCII code of a backslash (\), and - the next character is a dot (.), then CopyReadLineText function would incorrectly interpret the sequence as an end-of-copy marker (\.). This can only happen in encodings that can "embed" ascii characters as the second byte. One example of such sequence is '\x5ca45c2e666f6f' in Big5 encoding. If you put that in a file, and load it with COPY FROM, you'd incorrectly get an "end-of-copy marker corrupt" error. Backpatch to all supported versions. Reviewed-by: John Naylor, Kyotaro Horiguchi Discussion: https://www.postgresql.org/message-id/a897f84f-8dca-8798-3139-07da5bb38728%40iki.fi --- src/backend/commands/copyfromparse.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index b843d315b1..315b16fd7a 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -1084,7 +1084,7 @@ CopyReadLineText(CopyFromState cstate) break; } else if (!cstate->opts.csv_mode) - + { /* * If we are here, it means we found a backslash followed by * something other than a period. In non-CSV mode, anything @@ -1095,8 +1095,16 @@ CopyReadLineText(CopyFromState cstate) * backslashes are not special, so we want to process the * character after the backslash just like a normal character, * so we don't increment in those cases. + * + * Set 'c' to skip whole character correctly in multi-byte + * encodings. If we don't have the whole character in the + * buffer yet, we might loop back to process it, after all, + * but that's OK because multi-byte characters cannot have any + * special meaning. */ raw_buf_ptr++; + c = c2; + } } /* -- 2.30.0 --------------CFC80B40DFDD66DF067F288D-- ^ permalink raw reply [nested|flat] 45+ messages in thread
* Re: [BUG]Update Toast data failure in logical replication @ 2022-01-24 20:10 Robert Haas <[email protected]> 0 siblings, 0 replies; 45+ messages in thread From: Robert Haas @ 2022-01-24 20:10 UTC (permalink / raw) To: Amit Kapila <[email protected]>; +Cc: Dilip Kumar <[email protected]>; Petr Jelinek <[email protected]>; Peter Eisentraut <[email protected]>; Tomas Vondra <[email protected]>; Alvaro Herrera <[email protected]>; Andres Freund <[email protected]>; Kuntal Ghosh <[email protected]>; [email protected] <[email protected]>; pgsql-hackers On Tue, Aug 10, 2021 at 1:20 AM Amit Kapila <[email protected]> wrote: > It seems to me this problem exists from the time we introduced > wal_level = logical in the commit e55704d8b2 [1], or another > possibility is that logical replication commit didn't consider > something to make it work. Andres, Robert, Petr, can you guys please > comment because otherwise, we might miss something here. I'm belatedly getting around to looking at this thread. My recollection of this is: I think we realized when we were working on the logical decoding stuff that the key columns of the old tuple would have to be detoasted in order for the mechanism to work, because I remember worrying about whether it would potentially be a problem that the WAL record would end up huge. However, I think we believed that the new tuple wouldn't need to have the detoasted values, because logical decoding is designed to notice all the TOAST insertions for the new tuple and reassemble those separate chunks to get the original value back. And off-hand I'm not sure why that logic doesn't apply just as much to the key columns as any others. But the evidence does suggest that there's some kind of bug here, so evidently there's some flaw in that line of thinking. I'm not sure off-hand what it is, though. -- Robert Haas EDB: http://www.enterprisedb.com ^ permalink raw reply [nested|flat] 45+ messages in thread
end of thread, other threads:[~2022-01-24 20:10 UTC | newest] Thread overview: 45+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2021-02-03 12:00 [PATCH 1/1] Fix a corner-case in COPY FROM backslash processing. Heikki Linnakangas <[email protected]> 2021-02-03 12:00 [PATCH 1/1] Fix a corner-case in COPY FROM backslash processing. Heikki Linnakangas <[email protected]> 2021-02-03 12:00 [PATCH 1/1] Fix a corner-case in COPY FROM backslash processing. Heikki Linnakangas <[email protected]> 2021-02-03 12:00 [PATCH 1/1] Fix a corner-case in COPY FROM backslash processing. Heikki Linnakangas <[email protected]> 2021-02-03 12:00 [PATCH 1/1] Fix a corner-case in COPY FROM backslash processing. Heikki Linnakangas <[email protected]> 2021-02-03 12:00 [PATCH 1/1] Fix a corner-case in COPY FROM backslash processing. Heikki Linnakangas <[email protected]> 2021-02-03 12:00 [PATCH 1/1] Fix a corner-case in COPY FROM backslash processing. Heikki Linnakangas <[email protected]> 2021-02-03 12:00 [PATCH 1/1] Fix a corner-case in COPY FROM backslash processing. Heikki Linnakangas <[email protected]> 2021-02-03 12:00 [PATCH 1/1] Fix a corner-case in COPY FROM backslash processing. Heikki Linnakangas <[email protected]> 2021-02-03 12:00 [PATCH 1/1] Fix a corner-case in COPY FROM backslash processing. Heikki Linnakangas <[email protected]> 2021-02-03 12:00 [PATCH 1/1] Fix a corner-case in COPY FROM backslash processing. Heikki Linnakangas <[email protected]> 2021-02-03 12:00 [PATCH 1/1] Fix a corner-case in COPY FROM backslash processing. Heikki Linnakangas <[email protected]> 2021-02-03 12:00 [PATCH 1/1] Fix a corner-case in COPY FROM backslash processing. Heikki Linnakangas <[email protected]> 2021-02-03 12:00 [PATCH 1/1] Fix a corner-case in COPY FROM backslash processing. Heikki Linnakangas <[email protected]> 2021-02-03 12:00 [PATCH 1/1] Fix a corner-case in COPY FROM backslash processing. Heikki Linnakangas <[email protected]> 2021-02-03 12:00 [PATCH 1/1] Fix a corner-case in COPY FROM backslash processing. Heikki Linnakangas <[email protected]> 2021-02-03 12:00 [PATCH 1/1] Fix a corner-case in COPY FROM backslash processing. Heikki Linnakangas <[email protected]> 2021-02-03 12:00 [PATCH 1/1] Fix a corner-case in COPY FROM backslash processing. Heikki Linnakangas <[email protected]> 2021-02-03 12:00 [PATCH 1/1] Fix a corner-case in COPY FROM backslash processing. Heikki Linnakangas <[email protected]> 2021-02-03 12:00 [PATCH 1/1] Fix a corner-case in COPY FROM backslash processing. Heikki Linnakangas <[email protected]> 2021-02-03 12:00 [PATCH 1/1] Fix a corner-case in COPY FROM backslash processing. Heikki Linnakangas <[email protected]> 2021-02-03 12:00 [PATCH 1/1] Fix a corner-case in COPY FROM backslash processing. Heikki Linnakangas <[email protected]> 2021-02-04 19:36 [PATCH v2 1/1] Fix a corner-case in COPY FROM backslash processing. Heikki Linnakangas <[email protected]> 2021-02-04 19:36 [PATCH v2 1/1] Fix a corner-case in COPY FROM backslash processing. Heikki Linnakangas <[email protected]> 2021-02-04 19:36 [PATCH v2 1/1] Fix a corner-case in COPY FROM backslash processing. Heikki Linnakangas <[email protected]> 2021-02-04 19:36 [PATCH v2 1/1] Fix a corner-case in COPY FROM backslash processing. Heikki Linnakangas <[email protected]> 2021-02-04 19:36 [PATCH v2 1/1] Fix a corner-case in COPY FROM backslash processing. Heikki Linnakangas <[email protected]> 2021-02-04 19:36 [PATCH v2 1/1] Fix a corner-case in COPY FROM backslash processing. Heikki Linnakangas <[email protected]> 2021-02-04 19:36 [PATCH v2 1/1] Fix a corner-case in COPY FROM backslash processing. Heikki Linnakangas <[email protected]> 2021-02-04 19:36 [PATCH v2 1/1] Fix a corner-case in COPY FROM backslash processing. Heikki Linnakangas <[email protected]> 2021-02-04 19:36 [PATCH v2 1/1] Fix a corner-case in COPY FROM backslash processing. Heikki Linnakangas <[email protected]> 2021-02-04 19:36 [PATCH v2 1/1] Fix a corner-case in COPY FROM backslash processing. Heikki Linnakangas <[email protected]> 2021-02-04 19:36 [PATCH v2 1/1] Fix a corner-case in COPY FROM backslash processing. Heikki Linnakangas <[email protected]> 2021-02-04 19:36 [PATCH v2 1/1] Fix a corner-case in COPY FROM backslash processing. Heikki Linnakangas <[email protected]> 2021-02-04 19:36 [PATCH v2 1/1] Fix a corner-case in COPY FROM backslash processing. Heikki Linnakangas <[email protected]> 2021-02-04 19:36 [PATCH v2 1/1] Fix a corner-case in COPY FROM backslash processing. Heikki Linnakangas <[email protected]> 2021-02-04 19:36 [PATCH v2 1/1] Fix a corner-case in COPY FROM backslash processing. Heikki Linnakangas <[email protected]> 2021-02-04 19:36 [PATCH v2 1/1] Fix a corner-case in COPY FROM backslash processing. Heikki Linnakangas <[email protected]> 2021-02-04 19:36 [PATCH v2 1/1] Fix a corner-case in COPY FROM backslash processing. Heikki Linnakangas <[email protected]> 2021-02-04 19:36 [PATCH v2 1/1] Fix a corner-case in COPY FROM backslash processing. Heikki Linnakangas <[email protected]> 2021-02-04 19:36 [PATCH v2 1/1] Fix a corner-case in COPY FROM backslash processing. Heikki Linnakangas <[email protected]> 2021-02-04 19:36 [PATCH v2 1/1] Fix a corner-case in COPY FROM backslash processing. Heikki Linnakangas <[email protected]> 2021-02-04 19:36 [PATCH v2 1/1] Fix a corner-case in COPY FROM backslash processing. Heikki Linnakangas <[email protected]> 2021-02-04 19:36 [PATCH v2 1/1] Fix a corner-case in COPY FROM backslash processing. Heikki Linnakangas <[email protected]> 2022-01-24 20:10 Re: [BUG]Update Toast data failure in logical replication Robert Haas <[email protected]>
This inbox is served by agora; see mirroring instructions for how to clone and mirror all data and code used for this inbox