From: Heikki Linnakangas <hlinnaka@iki.fi>
To: wizard_1024@tut.by
To: pgsql-bugs@lists.postgresql.org
Subject: Re: BUG #17501: COPY is failing with "ERROR: invalid byte sequence for encoding "UTF8": 0xe5"
Date: Sun, 29 May 2022 13:39:52 +0300
Message-ID: <196eab40-0754-0fe5-1419-b50459fbb6fc@iki.fi> (raw)
In-Reply-To: <17501-128b1dd039362ae6@postgresql.org>
References: <17501-128b1dd039362ae6@postgresql.org>
On 28/05/2022 23:52, PG Bug reporting form wrote:
> The following bug has been logged on the website:
>
> Bug reference: 17501
> Logged by: Vitaly Voronov
> Email address: wizard_1024@tut.by
> PostgreSQL version: 14.3
> Operating system: CentOS Linux release 7.9.2009 (Core)
> Description:
>
> Hello,
>
> We've seen a such bug: COPY command shows error "ERROR: invalid byte
> sequence for encoding "UTF8": 0xe5" on file.
> The same file with small amount of lines is imported without any errors.
I was able to reproduce this with your instructions. Thank you!
The bug was indeed introduced by commit f82de5c46b, in v14. Attached
patch fixes it. Barring any objections, I'll commit and backpatch this.
I also noticed that there are some outdated comments in copyfromparse.c
that still talk about the old protocol version 2 and COPY_OLD_FE. I'll
clean that up as a separate commit.
- Heikki
Attachments:
[text/x-patch] 0001-Fix-COPY-FROM-when-database-encoding-is-SQL_ASCII.patch (1.4K, ../196eab40-0754-0fe5-1419-b50459fbb6fc@iki.fi/2-0001-Fix-COPY-FROM-when-database-encoding-is-SQL_ASCII.patch)
download | inline diff:
From c229e960125f363af52af97b66733c05b1495e87 Mon Sep 17 00:00:00 2001
From: Heikki Linnakangas <heikki.linnakangas@iki.fi>
Date: Sun, 29 May 2022 13:14:57 +0300
Subject: [PATCH 1/1] Fix COPY FROM when database encoding is SQL_ASCII.
In the codepath when no encoding conversion is required, the check for
incomplete character at the end of input incorrectly used server
encoding's max character length, instead of the client's. Usually the
server and client encodings are the same when we're not performing
encoding conversion, but SQL_ASCII is an exception.
Per bug #17501 from Vitaly Voronov. Backpatch to v14 where this was
introduced.
Discussion: https://www.postgresql.org/message-id/17501-128b1dd039362ae6@postgresql.org
---
src/backend/commands/copyfromparse.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.cindex edb80e2cd5..1cb889a7d9 100644--- a/src/backend/commands/copyfromparse.c+++ b/src/backend/commands/copyfromparse.c@@ -444,7 +444,7 @@ CopyConvertBuf(CopyFromState cstate)
* least one character, and a failure to do so means that we've
* hit an invalid byte sequence.
*/
- if (cstate->raw_reached_eof || unverifiedlen >= pg_database_encoding_max_length())+ if (cstate->raw_reached_eof || unverifiedlen >= pg_encoding_max_length(cstate->file_encoding))
cstate->input_reached_error = true;
return;
}
--
2.30.2
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-bugs@postgresql.org
Cc: hlinnaka@iki.fi, wizard_1024@tut.by, pgsql-bugs@lists.postgresql.org
Subject: Re: BUG #17501: COPY is failing with "ERROR: invalid byte sequence for encoding "UTF8": 0xe5"
In-Reply-To: <196eab40-0754-0fe5-1419-b50459fbb6fc@iki.fi>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox