From: Tom Lane <tgl@sss.pgh.pa.us>
To: Hugh Ranalli <hugh@whtc.ca>
Cc: Daniel Verite <daniel@manitou-mail.org>
Cc: Thomas Munro <thomas.munro@enterprisedb.com>
Cc: pgsql-bugs@lists.postgresql.org
Subject: Re: BUG #15548: Unaccent does not remove combining diacritical characters
Date: Sat, 15 Dec 2018 13:44:48 -0500
Message-ID: <23237.1544899488@sss.pgh.pa.us> (raw)
In-Reply-To: <CAAhbUMN1n=ZVns-OeCbaVRYPS0oj7tTnmJrzw7Az-op4DHC+JA@mail.gmail.com>
References: <10200.1544713542@sss.pgh.pa.us>
<5d77cc08-d582-4f83-a17f-f2c992d123a9@manitou-mail.org>
<CAAhbUMOHkoN3Jeti4dp1jz3VY=XZPcCqpX=sW=mgmJbdMS--ng@mail.gmail.com>
<CAAhbUMNqJXTN+_vYdi5L4CLjoq9OCG29V597RKrCQ7xKsCAejA@mail.gmail.com>
<16726.1544827803@sss.pgh.pa.us>
<CAAhbUMN1n=ZVns-OeCbaVRYPS0oj7tTnmJrzw7Az-op4DHC+JA@mail.gmail.com>
Hugh Ranalli <hugh@whtc.ca> writes:
> On Fri, 14 Dec 2018 at 17:50, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> Me too -- seems like that bears looking into. Perhaps the script's
>> results are platform dependent -- what were you testing on?
> I'm on Linux Mint 17, which is based on Ubuntu 14.04. But I don't think
> that's it. The program's decisions come from the two data files, the
> Unicode data set and the Latin-ASCII transliteration file. The script uses
> categories (
> ftp://ftp.unicode.org/Public/3.0-Update/UnicodeData-3.0.0.html#General%20Category)
> to identify letters (and now combining marks) and if they are in range,
> performs a substitution. It then uses the transliteration file to find
> rules for particular character substitutions (for example, that file seems
> to handle the copyright symbol substitution). I don't see anything platform
> dependent in there.
Hm. Something funny is going on here. When I fetch the two reference
files from the URLs cited in the script, and do
python2 generate_unaccent_rules.py --unicode-data-file UnicodeData.txt --latin-ascii-file Latin-ASCII.xml >newrules
I get something that's bit-for-bit the same as what's in unaccent.rules.
So there's clearly a platform difference between here and there.
I'm using Python 2.6.6, which is what ships with RHEL6; have not tried
it on anything newer.
regards, tom lane
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-bugs@postgresql.org
Cc: tgl@sss.pgh.pa.us, hugh@whtc.ca, daniel@manitou-mail.org, thomas.munro@enterprisedb.com, pgsql-bugs@lists.postgresql.org
Subject: Re: BUG #15548: Unaccent does not remove combining diacritical characters
In-Reply-To: <23237.1544899488@sss.pgh.pa.us>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox