Date: Tue, 12 Feb 2019 13:18:19 +0900
From: Michael Paquier <michael@paquier.xyz>
To: Ramanarayana <raam.soft@gmail.com>
Cc: Hugh Ranalli <hugh@whtc.ca>, pgsql-hackers@lists.postgresql.org
Subject: Re: BUG #15548: Unaccent does not remove combining diacritical
 characters
Message-ID: <20190212041819.GK1475@paquier.xyz>
References: <15548-cef1b3f8de190d4f@postgresql.org>
 <CAEepm=2qfx8Q9PEq3H5KJ74a6RDgpsaJf0kK+PyGEBneC0Sr1g@mail.gmail.com>
 <154982918542.11785.1374991294537224097.pgcf@coridan.postgresql.org>
 <CAAhbUMODj1cCHjCpZ-=kxJxnVWyTsqu6ZnWe8+gCsb5SGnv=zA@mail.gmail.com>
 <CAKm4Xs7CBuCW_XQtrVX6ThwSMiL0WK7Cj3nZx2Jymb9eJ=YdMQ@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha512;
	protocol="application/pgp-signature"; boundary="aziWXe2aaRGlkyg3"
Content-Disposition: inline
In-Reply-To: 
 <CAKm4Xs7CBuCW_XQtrVX6ThwSMiL0WK7Cj3nZx2Jymb9eJ=YdMQ@mail.gmail.com>
Precedence: bulk


--aziWXe2aaRGlkyg3
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On Tue, Feb 12, 2019 at 02:27:31AM +0530, Ramanarayana wrote:
> I tested the script in python 2.7 and it works perfect. The problem is in
> python 3.7(and may be only in windows as you were not getting the issue)
> and I was getting the following error
>=20
> UnicodeEncodeError: 'charmap' codec can't encode character '\u0100' in
> position 0: character maps to <undefined>
>=20
>  I went through the python script and found that the stdout encoding is s=
et
> to utf-8 only  if python version is <=3D2.
>=20
> I have made the same change for python version 3 as well. Please find the
> patch for the same.Let me know if it makes sense

Isn't that because Windows encoding becomes cp1252, utf16 or such?
FWIW, on Debian SID with Python 3.7, I get the correct output, and no
diffs on HEAD.  Perhaps it would make sense to use open() on the
different files with encoding=3D'utf-8' to avoid any kind of problems?
--
Michael

--aziWXe2aaRGlkyg3
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----

iQIzBAABCgAdFiEEG72nH6vTowiyblFKnvQgOdbyQH0FAlxiSQsACgkQnvQgOdby
QH1F/w//ZGOYAP9SLDjjpuOs0A7mWLwt8e2fADJ/vY0LlAIk7zFYyu35gfNioCHc
cQjCJm8moahzKlbziVJCr1IS3DsjnUVUD/+o9Bb5TQ8F/rDKsHivK5kztOOIYQ0+
GexUz8W+/Kxe5Xzhkx14TCBwzucykRmAZGxEQYorJzwKBJ2Fix3wveyvkuN5t3t7
G5KK0H4/R4fy5w6+XoF66LHVOA6MZWQwIxrnl3pV6OushX7ES+/fXGed79F3fcF/
s1ZpZez4CaNNKJ2kdUNltYVXz+fRfwfgFQnkxlJ3Emxwm1Fv2Z0kHxkKHoiKqQSN
pvd2S3snFKf3h7oeL1GpZ0V+Jc8FGhzqJpJY2PuWH9hnLmtlccq3w8z2qJzwIeRk
FqZVPe4IuSJtjsEPZu8ZCKg6bvYg1OfkjitsZcR2LRWsXDrUQmhzvP0zpK2Lv5El
8QgMMZa4CkB7DPMureRERY//FhY1NEyIuuiDYR5QlaHQMCmdPyXgVva5HxH8pWUD
von9ndll+Tm7mq5Qw/OeW/NDODcLCbNQxi6phQeb5kGFTKqGKY91yhJNbYAtdKYT
QuywBuuHpVSejpW2Q0qV8gROGKkzxTbfYgPpAJDX2+ZzenhkWcH18wI1Kxq/MYJc
/32T5ddTh1nWNKTqeZsEvcgEpZ1wjvAz8ZroYf8+dW8PKnUqRyI=
=csMp
-----END PGP SIGNATURE-----

--aziWXe2aaRGlkyg3--