From: Tom Lane <tgl@sss.pgh.pa.us>
To: Michael Paquier <michael@paquier.xyz>
Cc: Thomas Munro <thomas.munro@enterprisedb.com>
Cc: hugh@whtc.ca, daniel@manitou-mail.org, PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
Subject: Re: BUG #15548: Unaccent does not remove combining diacritical characters
Date: Tue, 18 Dec 2018 01:23:57 -0500
Message-ID: <11345.1545114237@sss.pgh.pa.us> (raw)
In-Reply-To: <20181218060735.GL1532@paquier.xyz>
References: <CAAhbUMO+fTZTwigDJ=tB3qFvHMe-xAJO5QpsFPH6Vb2oDYAU6w@mail.gmail.com>
<CAAhbUMMmXnj0YSD+fr5hSqeC+D6PAG+0kXJwMMhK2DCdwQVoxQ@mail.gmail.com>
<29419.1544908811@sss.pgh.pa.us>
<CAEepm=3gSmNWkteBxCEL-W+j1dmbcNzDin_iv+f_Om6o+1fAiA@mail.gmail.com>
<CAAhbUMOX4QLj6c0O3GnjZYtR2dpAowss832Bq1n7oJyByeR7kQ@mail.gmail.com>
<31564.1545078667@sss.pgh.pa.us>
<CAAhbUMOe_VeJBxVqPe4Op2XKrT+xCcCvsUYRH4v6G71NDGG=fg@mail.gmail.com>
<CAEepm=0qb_nx-f8cACS1=1NdmCj-3D9zXFU+RJHsFbZEztcqjg@mail.gmail.com>
<20181218045708.GI1532@paquier.xyz>
<8506.1545111362@sss.pgh.pa.us>
<20181218060735.GL1532@paquier.xyz>
Michael Paquier <michael@paquier.xyz> writes:
> On Tue, Dec 18, 2018 at 12:36:02AM -0500, Tom Lane wrote:
>> tl;dr: I think we should convert unaccent.sql and unaccent.out
>> to UTF8 encoding. Then, adding more test cases for this patch
>> will be easy.
> Do you think that we could also remove the non-ASCII characters from the
> tests? It would be easy enough to use E'\xNN' (utf8 hex) or such in
> input, and show the output with bytea.
I'm not really for that, because it would make the test cases harder
to verify by eyeball. With the current setup --- other than the
uncommon-outside-Russia encoding choice --- you don't really need
to read or speak Russian to see that this:
SELECT unaccent('ёлка');
unaccent
----------
елка
(1 row)
probably represents unaccent doing what it ought to. If everything
is in hex then it's a lot harder.
Ten years ago I might've agreed with your point, but today it's
hard to believe that anyone who takes any interest at all in
unaccent's functionality would not have a UTF8-capable terminal.
> That's harder to read, still we
> discussed about not using UTF-8 in the python script to allow folks with
> simple terminals to touch the code the last time this was touched
> (5e8d670) and the characters used could be documented as comments in the
> tests.
Maybe I'm misremembering, but I thought that discussion was about the
code files. I am still mistrustful of non-ASCII in our code files.
But for data and test files, we've been accepting UTF8 ever since the
text-search-in-core stuff landed. Heck, unaccent.rules itself is UTF8.
regards, tom lane
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-bugs@postgresql.org
Cc: tgl@sss.pgh.pa.us, michael@paquier.xyz, thomas.munro@enterprisedb.com, pgsql-bugs@lists.postgresql.org
Subject: Re: BUG #15548: Unaccent does not remove combining diacritical characters
In-Reply-To: <11345.1545114237@sss.pgh.pa.us>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox