Content-Type: multipart/mixed; boundary="------------nlwXVy0cvr4bSSMHn493UtIK"
Message-ID: <196eab40-0754-0fe5-1419-b50459fbb6fc@iki.fi>
Date: Sun, 29 May 2022 13:39:52 +0300
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101
 Thunderbird/91.9.0
Subject: Re: BUG #17501: COPY is failing with "ERROR: invalid byte sequence
 for encoding "UTF8": 0xe5"
Content-Language: en-US
To: wizard_1024@tut.by, pgsql-bugs@lists.postgresql.org
References: <17501-128b1dd039362ae6@postgresql.org>
From: Heikki Linnakangas <hlinnaka@iki.fi>
In-Reply-To: <17501-128b1dd039362ae6@postgresql.org>
Archived-At: 
 <https://www.postgresql.org/message-id/196eab40-0754-0fe5-1419-b50459fbb6fc%40iki.fi>
Precedence: bulk

This is a multi-part message in MIME format.
--------------nlwXVy0cvr4bSSMHn493UtIK
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit

On 28/05/2022 23:52, PG Bug reporting form wrote:
> The following bug has been logged on the website:
> 
> Bug reference:      17501
> Logged by:          Vitaly Voronov
> Email address:      wizard_1024@tut.by
> PostgreSQL version: 14.3
> Operating system:   CentOS Linux release 7.9.2009 (Core)
> Description:
> 
> Hello,
> 
> We've seen a such bug: COPY command shows error "ERROR:  invalid byte
> sequence for encoding "UTF8": 0xe5" on file.
> The same file with small amount of lines is imported without any errors.

I was able to reproduce this with your instructions. Thank you!

The bug was indeed introduced by commit f82de5c46b, in v14. Attached 
patch fixes it. Barring any objections, I'll commit and backpatch this.

I also noticed that there are some outdated comments in copyfromparse.c 
that still talk about the old protocol version 2 and COPY_OLD_FE. I'll 
clean that up as a separate commit.

- Heikki
--------------nlwXVy0cvr4bSSMHn493UtIK
Content-Type: text/x-patch; charset=UTF-8;
 name="0001-Fix-COPY-FROM-when-database-encoding-is-SQL_ASCII.patch"
Content-Disposition: attachment;
 filename*0="0001-Fix-COPY-FROM-when-database-encoding-is-SQL_ASCII.patch"
Content-Transfer-Encoding: base64

RnJvbSBjMjI5ZTk2MDEyNWYzNjNhZjUyYWY5N2I2NjczM2MwNWIxNDk1ZTg3IE1vbiBTZXAg
MTcgMDA6MDA6MDAgMjAwMQpGcm9tOiBIZWlra2kgTGlubmFrYW5nYXMgPGhlaWtraS5saW5u
YWthbmdhc0Bpa2kuZmk+CkRhdGU6IFN1biwgMjkgTWF5IDIwMjIgMTM6MTQ6NTcgKzAzMDAK
U3ViamVjdDogW1BBVENIIDEvMV0gRml4IENPUFkgRlJPTSB3aGVuIGRhdGFiYXNlIGVuY29k
aW5nIGlzIFNRTF9BU0NJSS4KCkluIHRoZSBjb2RlcGF0aCB3aGVuIG5vIGVuY29kaW5nIGNv
bnZlcnNpb24gaXMgcmVxdWlyZWQsIHRoZSBjaGVjayBmb3IKaW5jb21wbGV0ZSBjaGFyYWN0
ZXIgYXQgdGhlIGVuZCBvZiBpbnB1dCBpbmNvcnJlY3RseSB1c2VkIHNlcnZlcgplbmNvZGlu
ZydzIG1heCBjaGFyYWN0ZXIgbGVuZ3RoLCBpbnN0ZWFkIG9mIHRoZSBjbGllbnQncy4gVXN1
YWxseSB0aGUKc2VydmVyIGFuZCBjbGllbnQgZW5jb2RpbmdzIGFyZSB0aGUgc2FtZSB3aGVu
IHdlJ3JlIG5vdCBwZXJmb3JtaW5nCmVuY29kaW5nIGNvbnZlcnNpb24sIGJ1dCBTUUxfQVND
SUkgaXMgYW4gZXhjZXB0aW9uLgoKUGVyIGJ1ZyAjMTc1MDEgZnJvbSBWaXRhbHkgVm9yb25v
di4gQmFja3BhdGNoIHRvIHYxNCB3aGVyZSB0aGlzIHdhcwppbnRyb2R1Y2VkLgoKRGlzY3Vz
c2lvbjogaHR0cHM6Ly93d3cucG9zdGdyZXNxbC5vcmcvbWVzc2FnZS1pZC8xNzUwMS0xMjhi
MWRkMDM5MzYyYWU2QHBvc3RncmVzcWwub3JnCi0tLQogc3JjL2JhY2tlbmQvY29tbWFuZHMv
Y29weWZyb21wYXJzZS5jIHwgMiArLQogMSBmaWxlIGNoYW5nZWQsIDEgaW5zZXJ0aW9uKCsp
LCAxIGRlbGV0aW9uKC0pCgpkaWZmIC0tZ2l0IGEvc3JjL2JhY2tlbmQvY29tbWFuZHMvY29w
eWZyb21wYXJzZS5jIGIvc3JjL2JhY2tlbmQvY29tbWFuZHMvY29weWZyb21wYXJzZS5jCmlu
ZGV4IGVkYjgwZTJjZDUuLjFjYjg4OWE3ZDkgMTAwNjQ0Ci0tLSBhL3NyYy9iYWNrZW5kL2Nv
bW1hbmRzL2NvcHlmcm9tcGFyc2UuYworKysgYi9zcmMvYmFja2VuZC9jb21tYW5kcy9jb3B5
ZnJvbXBhcnNlLmMKQEAgLTQ0NCw3ICs0NDQsNyBAQCBDb3B5Q29udmVydEJ1ZihDb3B5RnJv
bVN0YXRlIGNzdGF0ZSkKIAkJCSAqIGxlYXN0IG9uZSBjaGFyYWN0ZXIsIGFuZCBhIGZhaWx1
cmUgdG8gZG8gc28gbWVhbnMgdGhhdCB3ZSd2ZQogCQkJICogaGl0IGFuIGludmFsaWQgYnl0
ZSBzZXF1ZW5jZS4KIAkJCSAqLwotCQkJaWYgKGNzdGF0ZS0+cmF3X3JlYWNoZWRfZW9mIHx8
IHVudmVyaWZpZWRsZW4gPj0gcGdfZGF0YWJhc2VfZW5jb2RpbmdfbWF4X2xlbmd0aCgpKQor
CQkJaWYgKGNzdGF0ZS0+cmF3X3JlYWNoZWRfZW9mIHx8IHVudmVyaWZpZWRsZW4gPj0gcGdf
ZW5jb2RpbmdfbWF4X2xlbmd0aChjc3RhdGUtPmZpbGVfZW5jb2RpbmcpKQogCQkJCWNzdGF0
ZS0+aW5wdXRfcmVhY2hlZF9lcnJvciA9IHRydWU7CiAJCQlyZXR1cm47CiAJCX0KLS0gCjIu
MzAuMgoK

--------------nlwXVy0cvr4bSSMHn493UtIK--