pg.ddx.io pgsql-bugs@postgresql.org mailing list archive
help / color / mirror / Atom feedBUG #17501: COPY is failing with "ERROR: invalid byte sequence for encoding "UTF8": 0xe5"
5+ messages / 3 participants
[nested] [flat]
* BUG #17501: COPY is failing with "ERROR: invalid byte sequence for encoding "UTF8": 0xe5"
@ 2022-05-28 20:52 PG Bug reporting form <noreply@postgresql.org>
0 siblings, 2 replies; 5+ messages in thread
From: PG Bug reporting form @ 2022-05-28 20:52 UTC (permalink / raw)
To: pgsql-bugs@lists.postgresql.org; +Cc: wizard_1024@tut.by
The following bug has been logged on the website:
Bug reference: 17501
Logged by: Vitaly Voronov
Email address: wizard_1024@tut.by
PostgreSQL version: 14.3
Operating system: CentOS Linux release 7.9.2009 (Core)
Description:
Hello,
We've seen a such bug: COPY command shows error "ERROR: invalid byte
sequence for encoding "UTF8": 0xe5" on file.
The same file with small amount of lines is imported without any errors.
How to reproduce bug:
# create database
# create database with
# SQL_ASCII, C, C
createdb --encoding=SQL_ASCII --lc-collate=C --lc-ctype=C
--template=template0 test
# connect to the database
psql test
# Create table
CREATE TABLE test_data (
test_data text
);
# Import without error
truncate table test_data;
COPY test_data (test_data) FROM '/tmp/test_pass.csv' WITH DELIMITER AS ','
CSV QUOTE AS '"';
COPY 207
# Import with error
truncate table test_data;
COPY test_data (test_data) FROM '/tmp/test_fail.csv' WITH DELIMITER AS ','
CSV QUOTE AS '"';
ERROR: invalid byte sequence for encoding "UTF8": 0xe5
CONTEXT: COPY test_data, line 627
# both files contains the same rows, but test_fail contains more rows
# seems that the file more than 65K size cannot be imported
# if create DB with UTF8 encoding instead of SQL_ASCII - both tests will be
passed
# How to generate files:
# Imported without errors
for i in $(seq 1 207); do echo
"NURO光です。明日の宅内工事お立合いよろしくお願い致します。2回目の屋外工事につきましては具体的な工事日案内の準備が整い次第、こちらからご連絡いたします。※詳細はこちら【工事について】https://www.test.jp/1234/5678.html&id=12211";
>> /tmp/test_pass.csv; done;
# Imported with errors
for i in $(seq 1 5722); do echo
"NURO光です。明日の宅内工事お立合いよろしくお願い致します。2回目の屋外工事につきましては具体的な工事日案内の準備が整い次第、こちらからご連絡いたします。※詳細はこちら【工事について】https://www.test.jp/1234/5678.html&id=12211";
>> /tmp/test_fail.csv; done;
# Both files can be imported without any problem to PostgreSQL 11.
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: BUG #17501: COPY is failing with "ERROR: invalid byte sequence for encoding "UTF8": 0xe5"
@ 2022-05-28 20:57 Vitaly V. Voronov <wizard_1024@tut.by>
parent: PG Bug reporting form <noreply@postgresql.org>
1 sibling, 1 reply; 5+ messages in thread
From: Vitaly V. Voronov @ 2022-05-28 20:57 UTC (permalink / raw)
To: pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
<div>Hello,</div><div> </div><div>Right commands:</div><div><div><span style="background-color:#ffffff;color:#000000;float:none;font-family:'ys text' , 'arial' , sans-serif;font-size:16px;font-style:normal;font-weight:400;text-decoration-style:initial;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"># Imported without errors</span><br style="background-color:rgb( 255 , 255 , 255 );color:rgb( 0 , 0 , 0 );font-family:'ys text' , 'arial' , sans-serif;font-size:16px;font-style:normal;font-weight:400;text-decoration-style:initial;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" /><span style="background-color:#ffffff;color:#000000;float:none;font-family:'ys text' , 'arial' , sans-serif;font-size:16px;font-style:normal;font-weight:400;text-decoration-style:initial;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">for i in $(seq 1 207); do echo</span><br style="background-color:rgb( 255 , 255 , 255 );color:rgb( 0 , 0 , 0 );font-family:'ys text' , 'arial' , sans-serif;font-size:16px;font-style:normal;font-weight:400;text-decoration-style:initial;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" /><span style="background-color:#ffffff;color:#000000;float:none;font-family:'ys text' , 'arial' , sans-serif;font-size:16px;font-style:normal;font-weight:400;text-decoration-style:initial;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">"NURO光です。明日の宅内工事お立合いよろしくお願い致します。2回目の屋外工事につきましては具体的な工事日案内の準備が整い次第、こちらからご連絡いたします。※詳細はこちら【工事について】</span><a href="https://www.test.jp/1234/5678.html&id=12211"; rel="noopener noreferrer" target="_blank" style="background-color:rgb( 255 , 255 , 255 );font-family:'ys text' , 'arial' , sans-serif;font-size:16px;font-style:normal;font-weight:400;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">https://www.test.jp/1234/5678.html&id=12211</a><span style="background-color:#ffffff;color:#000000;float:none;font-family:'ys text' , 'arial' , sans-serif;font-size:16px;font-style:normal;font-weight:400;text-decoration-style:initial;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">" </span>>> /tmp/test_pass.csv; done;</div></div><div><div><span style="background-color:#ffffff;color:#000000;float:none;font-family:'ys text' , 'arial' , sans-serif;font-size:16px;font-style:normal;font-weight:400;text-decoration-style:initial;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"># Imported with errors</span></div></div><div><div>for i in $(seq 1 5722); do echo "NURO光です。明日の宅内工事お立合いよろしくお願い致します。2回目の屋外工事につきましては具体的な工事日案内の準備が整い次第、こちらからご連絡いたします。※詳細はこちら【工事について】https://www.test.jp/1234/5678.html&id=12211"; >> /tmp/test_fail.csv; done;</div></div><div><br /></div><div><br /></div><div>28.05.2022, 23:53, "PG Bug reporting form" <noreply@postgresql.org>:</div><blockquote><p>The following bug has been logged on the website:<br /><br />Bug reference: 17501<br />Logged by: Vitaly Voronov<br />Email address: <a href="mailto:wizard_1024@tut.by">wizard_1024@tut.by</a><br />PostgreSQL version: 14.3<br />Operating system: CentOS Linux release 7.9.2009 (Core)<br />Description: <br /><br />Hello,<br /><br />We've seen a such bug: COPY command shows error "ERROR: invalid byte<br />sequence for encoding "UTF8": 0xe5" on file.<br />The same file with small amount of lines is imported without any errors.<br /><br />How to reproduce bug:<br /># create database<br /># create database with<br /># SQL_ASCII, C, C<br />createdb --encoding=SQL_ASCII --lc-collate=C --lc-ctype=C<br />--template=template0 test<br /><br /># connect to the database<br />psql test<br /><br /># Create table<br />CREATE TABLE test_data (<br /> test_data text<br />);<br /><br /># Import without error<br />truncate table test_data;<br />COPY test_data (test_data) FROM '/tmp/test_pass.csv' WITH DELIMITER AS ','<br />CSV QUOTE AS '"';<br /><br />COPY 207<br /><br /># Import with error<br />truncate table test_data;<br />COPY test_data (test_data) FROM '/tmp/test_fail.csv' WITH DELIMITER AS ','<br />CSV QUOTE AS '"';<br /><br />ERROR: invalid byte sequence for encoding "UTF8": 0xe5<br />CONTEXT: COPY test_data, line 627<br /><br /># both files contains the same rows, but test_fail contains more rows<br /># seems that the file more than 65K size cannot be imported<br /># if create DB with UTF8 encoding instead of SQL_ASCII - both tests will be<br />passed<br /><br /># How to generate files:<br /># Imported without errors<br />for i in $(seq 1 207); do echo<br />"NURO光です。明日の宅内工事お立合いよろしくお願い致します。2回目の屋外工事につきましては具体的な工事日案内の準備が整い次第、こちらからご連絡いたします。※詳細はこちら【工事について】<a href="https://www.test.jp/1234/5678.html&id=12211">https://www.test.jp/1234/5678.html&a... /></p><blockquote class="210e7a848e8fcb45wmi-quote"><blockquote class="210e7a848e8fcb45wmi-quote"> /tmp/test_pass.csv; done;<br /></blockquote></blockquote><p># Imported with errors<br />for i in $(seq 1 5722); do echo<br />"NURO光です。明日の宅内工事お立合いよろしくお願い致します。2回目の屋外工事につきましては具体的な工事日案内の準備が整い次第、こちらからご連絡いたします。※詳細はこちら【工事について】<a href="https://www.test.jp/1234/5678.html&id=12211">https://www.test.jp/1234/5678.html&a... /></p><blockquote class="210e7a848e8fcb45wmi-quote"><blockquote class="210e7a848e8fcb45wmi-quote"> /tmp/test_fail.csv; done;<br /></blockquote></blockquote><p><br /># Both files can be imported without any problem to PostgreSQL 11.<br /></p></blockquote>
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: BUG #17501: COPY is failing with "ERROR: invalid byte sequence for encoding "UTF8": 0xe5"
@ 2022-05-29 09:19 Heikki Linnakangas <hlinnaka@iki.fi>
parent: Vitaly V. Voronov <wizard_1024@tut.by>
0 siblings, 0 replies; 5+ messages in thread
From: Heikki Linnakangas @ 2022-05-29 09:19 UTC (permalink / raw)
To: Vitaly V. Voronov <wizard_1024@tut.by>; +Cc: pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
Sounds a lot like a bug in commit
f82de5c46bdf8cd65812a7b04c9509c218e1545d. Thanks for the report, I'll
investigate!
- Heikki
On 28/05/2022 23:57, Vitaly V. Voronov wrote:
> Hello,
> Right commands:
> # Imported without errors
> for i in $(seq 1 207); do echo
> "NURO光です。明日の宅内工事お立合いよろしくお願い致します。2回目の屋外工
> 事につきましては具体的な工事日案内の準備が整い次第、こちらからご連絡いた
> します。※詳細はこちら【工事について】https://www.test.jp/1234
> /5678.html&id=12211 <https://www.test.jp/1234/5678.html&id=12211>"; >>
> /tmp/test_pass.csv; done;
> # Imported with errors
> for i in $(seq 1 5722); do echo "NURO光です。明日の宅内工事お立合いよろ
> しくお願い致します。2回目の屋外工事につきましては具体的な工事日案内の準
> 備が整い次第、こちらからご連絡いたします。※詳細はこちら【工事について】
> https://www.test.jp/1234/5678.html&id=12211"; >> /tmp/test_fail.csv; done;
>
>
> 28.05.2022, 23:53, "PG Bug reporting form" <noreply@postgresql.org>:
>
> The following bug has been logged on the website:
>
> Bug reference: 17501
> Logged by: Vitaly Voronov
> Email address: wizard_1024@tut.by <mailto:wizard_1024@tut.by>
> PostgreSQL version: 14.3
> Operating system: CentOS Linux release 7.9.2009 (Core)
> Description:
>
> Hello,
>
> We've seen a such bug: COPY command shows error "ERROR: invalid byte
> sequence for encoding "UTF8": 0xe5" on file.
> The same file with small amount of lines is imported without any errors.
>
> How to reproduce bug:
> # create database
> # create database with
> # SQL_ASCII, C, C
> createdb --encoding=SQL_ASCII --lc-collate=C --lc-ctype=C
> --template=template0 test
>
> # connect to the database
> psql test
>
> # Create table
> CREATE TABLE test_data (
> test_data text
> );
>
> # Import without error
> truncate table test_data;
> COPY test_data (test_data) FROM '/tmp/test_pass.csv' WITH DELIMITER
> AS ','
> CSV QUOTE AS '"';
>
> COPY 207
>
> # Import with error
> truncate table test_data;
> COPY test_data (test_data) FROM '/tmp/test_fail.csv' WITH DELIMITER
> AS ','
> CSV QUOTE AS '"';
>
> ERROR: invalid byte sequence for encoding "UTF8": 0xe5
> CONTEXT: COPY test_data, line 627
>
> # both files contains the same rows, but test_fail contains more rows
> # seems that the file more than 65K size cannot be imported
> # if create DB with UTF8 encoding instead of SQL_ASCII - both tests
> will be
> passed
>
> # How to generate files:
> # Imported without errors
> for i in $(seq 1 207); do echo
> "NURO光です。明日の宅内工事お立合いよろしくお願い致します。2回目の屋
> 外工事につきましては具体的な工事日案内の準備が整い次第、こちらからご
> 連絡いたします。※詳細はこちら【工事について】https://www.test.jp
> /1234/5678.html&id=12211 <https://www.test.jp/1234/5678.html&id=12211>";
>
> /tmp/test_pass.csv; done;
>
> # Imported with errors
> for i in $(seq 1 5722); do echo
> "NURO光です。明日の宅内工事お立合いよろしくお願い致します。2回目の屋
> 外工事につきましては具体的な工事日案内の準備が整い次第、こちらからご
> 連絡いたします。※詳細はこちら【工事について】https://www.test.jp
> /1234/5678.html&id=12211 <https://www.test.jp/1234/5678.html&id=12211>";
>
> /tmp/test_fail.csv; done;
>
>
> # Both files can be imported without any problem to PostgreSQL 11.
>
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: BUG #17501: COPY is failing with "ERROR: invalid byte sequence for encoding "UTF8": 0xe5"
@ 2022-05-29 10:39 Heikki Linnakangas <hlinnaka@iki.fi>
parent: PG Bug reporting form <noreply@postgresql.org>
1 sibling, 1 reply; 5+ messages in thread
From: Heikki Linnakangas @ 2022-05-29 10:39 UTC (permalink / raw)
To: wizard_1024@tut.by; pgsql-bugs@lists.postgresql.org
On 28/05/2022 23:52, PG Bug reporting form wrote:
> The following bug has been logged on the website:
>
> Bug reference: 17501
> Logged by: Vitaly Voronov
> Email address: wizard_1024@tut.by
> PostgreSQL version: 14.3
> Operating system: CentOS Linux release 7.9.2009 (Core)
> Description:
>
> Hello,
>
> We've seen a such bug: COPY command shows error "ERROR: invalid byte
> sequence for encoding "UTF8": 0xe5" on file.
> The same file with small amount of lines is imported without any errors.
I was able to reproduce this with your instructions. Thank you!
The bug was indeed introduced by commit f82de5c46b, in v14. Attached
patch fixes it. Barring any objections, I'll commit and backpatch this.
I also noticed that there are some outdated comments in copyfromparse.c
that still talk about the old protocol version 2 and COPY_OLD_FE. I'll
clean that up as a separate commit.
- Heikki
Attachments:
[text/x-patch] 0001-Fix-COPY-FROM-when-database-encoding-is-SQL_ASCII.patch (1.4K, ../../196eab40-0754-0fe5-1419-b50459fbb6fc@iki.fi/2-0001-Fix-COPY-FROM-when-database-encoding-is-SQL_ASCII.patch)
download | inline diff:
From c229e960125f363af52af97b66733c05b1495e87 Mon Sep 17 00:00:00 2001
From: Heikki Linnakangas <heikki.linnakangas@iki.fi>
Date: Sun, 29 May 2022 13:14:57 +0300
Subject: [PATCH 1/1] Fix COPY FROM when database encoding is SQL_ASCII.
In the codepath when no encoding conversion is required, the check for
incomplete character at the end of input incorrectly used server
encoding's max character length, instead of the client's. Usually the
server and client encodings are the same when we're not performing
encoding conversion, but SQL_ASCII is an exception.
Per bug #17501 from Vitaly Voronov. Backpatch to v14 where this was
introduced.
Discussion: https://www.postgresql.org/message-id/17501-128b1dd039362ae6@postgresql.org
---
src/backend/commands/copyfromparse.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index edb80e2cd5..1cb889a7d9 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -444,7 +444,7 @@ CopyConvertBuf(CopyFromState cstate)
* least one character, and a failure to do so means that we've
* hit an invalid byte sequence.
*/
- if (cstate->raw_reached_eof || unverifiedlen >= pg_database_encoding_max_length())
+ if (cstate->raw_reached_eof || unverifiedlen >= pg_encoding_max_length(cstate->file_encoding))
cstate->input_reached_error = true;
return;
}
--
2.30.2
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: BUG #17501: COPY is failing with "ERROR: invalid byte sequence for encoding "UTF8": 0xe5"
@ 2022-05-29 21:09 Heikki Linnakangas <hlinnaka@iki.fi>
parent: Heikki Linnakangas <hlinnaka@iki.fi>
0 siblings, 0 replies; 5+ messages in thread
From: Heikki Linnakangas @ 2022-05-29 21:09 UTC (permalink / raw)
To: wizard_1024@tut.by; +Cc: pgsql-bugs@lists.postgresql.org
On 29/05/2022 13:39, Heikki Linnakangas wrote:
> On 28/05/2022 23:52, PG Bug reporting form wrote:
>> The following bug has been logged on the website:
>>
>> Bug reference: 17501
>> Logged by: Vitaly Voronov
>> Email address: wizard_1024@tut.by
>> PostgreSQL version: 14.3
>> Operating system: CentOS Linux release 7.9.2009 (Core)
>> Description:
>>
>> Hello,
>>
>> We've seen a such bug: COPY command shows error "ERROR: invalid byte
>> sequence for encoding "UTF8": 0xe5" on file.
>> The same file with small amount of lines is imported without any errors.
>
> I was able to reproduce this with your instructions. Thank you!
>
> The bug was indeed introduced by commit f82de5c46b, in v14. Attached
> patch fixes it. Barring any objections, I'll commit and backpatch this.
>
> I also noticed that there are some outdated comments in copyfromparse.c
> that still talk about the old protocol version 2 and COPY_OLD_FE. I'll
> clean that up as a separate commit.
Committed. And I included those comment changes in the same commit,
after all.
Thanks for the report!
- Heikki
^ permalink raw reply [nested|flat] 5+ messages in thread
end of thread, other threads:[~2022-05-29 21:09 UTC | newest]
Thread overview: 5+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2022-05-28 20:52 BUG #17501: COPY is failing with "ERROR: invalid byte sequence for encoding "UTF8": 0xe5" PG Bug reporting form <noreply@postgresql.org>
2022-05-28 20:57 ` Vitaly V. Voronov <wizard_1024@tut.by>
2022-05-29 09:19 ` Re: BUG #17501: COPY is failing with "ERROR: invalid byte sequence for encoding "UTF8": 0xe5" Heikki Linnakangas <hlinnaka@iki.fi>
2022-05-29 10:39 ` Re: BUG #17501: COPY is failing with "ERROR: invalid byte sequence for encoding "UTF8": 0xe5" Heikki Linnakangas <hlinnaka@iki.fi>
2022-05-29 21:09 ` Re: BUG #17501: COPY is failing with "ERROR: invalid byte sequence for encoding "UTF8": 0xe5" Heikki Linnakangas <hlinnaka@iki.fi>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox