From: Michael Paquier <michael@paquier.xyz>
To: Ramanarayana <raam.soft@gmail.com>
Cc: Hugh Ranalli <hugh@whtc.ca>
Cc: pgsql-hackers@lists.postgresql.org
Subject: Re: BUG #15548: Unaccent does not remove combining diacritical characters
Date: Tue, 12 Feb 2019 13:18:19 +0900
Message-ID: <20190212041819.GK1475@paquier.xyz> (raw)
In-Reply-To: <CAKm4Xs7CBuCW_XQtrVX6ThwSMiL0WK7Cj3nZx2Jymb9eJ=YdMQ@mail.gmail.com>
References: <15548-cef1b3f8de190d4f@postgresql.org>
<CAEepm=2qfx8Q9PEq3H5KJ74a6RDgpsaJf0kK+PyGEBneC0Sr1g@mail.gmail.com>
<154982918542.11785.1374991294537224097.pgcf@coridan.postgresql.org>
<CAAhbUMODj1cCHjCpZ-=kxJxnVWyTsqu6ZnWe8+gCsb5SGnv=zA@mail.gmail.com>
<CAKm4Xs7CBuCW_XQtrVX6ThwSMiL0WK7Cj3nZx2Jymb9eJ=YdMQ@mail.gmail.com>
On Tue, Feb 12, 2019 at 02:27:31AM +0530, Ramanarayana wrote:
> I tested the script in python 2.7 and it works perfect. The problem is in
> python 3.7(and may be only in windows as you were not getting the issue)
> and I was getting the following error
>
> UnicodeEncodeError: 'charmap' codec can't encode character '\u0100' in
> position 0: character maps to <undefined>
>
> I went through the python script and found that the stdout encoding is set
> to utf-8 only if python version is <=2.
>
> I have made the same change for python version 3 as well. Please find the
> patch for the same.Let me know if it makes sense
Isn't that because Windows encoding becomes cp1252, utf16 or such?
FWIW, on Debian SID with Python 3.7, I get the correct output, and no
diffs on HEAD. Perhaps it would make sense to use open() on the
different files with encoding='utf-8' to avoid any kind of problems?
--
Michael
Attachments:
[application/pgp-signature] signature.asc (832B, ../20190212041819.GK1475@paquier.xyz/2-signature.asc)
download
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-bugs@postgresql.org
Cc: michael@paquier.xyz, raam.soft@gmail.com, hugh@whtc.ca, pgsql-hackers@lists.postgresql.org
Subject: Re: BUG #15548: Unaccent does not remove combining diacritical characters
In-Reply-To: <20190212041819.GK1475@paquier.xyz>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox