From: Tom Lane <tgl@sss.pgh.pa.us>
To: Hugh Ranalli <hugh@whtc.ca>
cc: Daniel Verite <daniel@manitou-mail.org>,
        Thomas Munro <thomas.munro@enterprisedb.com>,
        pgsql-bugs@lists.postgresql.org
Subject: Re: BUG #15548: Unaccent does not remove combining diacritical
 characters
In-reply-to: 
 <CAAhbUMN1n=ZVns-OeCbaVRYPS0oj7tTnmJrzw7Az-op4DHC+JA@mail.gmail.com>
References: <10200.1544713542@sss.pgh.pa.us>
 <5d77cc08-d582-4f83-a17f-f2c992d123a9@manitou-mail.org>
 <CAAhbUMOHkoN3Jeti4dp1jz3VY=XZPcCqpX=sW=mgmJbdMS--ng@mail.gmail.com>
 <CAAhbUMNqJXTN+_vYdi5L4CLjoq9OCG29V597RKrCQ7xKsCAejA@mail.gmail.com>
 <16726.1544827803@sss.pgh.pa.us>
 <CAAhbUMN1n=ZVns-OeCbaVRYPS0oj7tTnmJrzw7Az-op4DHC+JA@mail.gmail.com>
Comments: In-reply-to Hugh Ranalli <hugh@whtc.ca>
	message dated "Sat, 15 Dec 2018 13:08:00 -0500"
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-ID: <23236.1544899488.1@sss.pgh.pa.us>
Content-Transfer-Encoding: quoted-printable
Date: Sat, 15 Dec 2018 13:44:48 -0500
Message-ID: <23237.1544899488@sss.pgh.pa.us>
Precedence: bulk

Hugh Ranalli <hugh@whtc.ca> writes:
> On Fri, 14 Dec 2018 at 17:50, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> Me too -- seems like that bears looking into.  Perhaps the script's
>> results are platform dependent -- what were you testing on?

> I'm on Linux Mint 17, which is based on Ubuntu 14.04. But I don't think
> that's it. The program's decisions come from the two data files, the
> Unicode data set and the Latin-ASCII transliteration file. The script us=
es
> categories (
> ftp://ftp.unicode.org/Public/3.0-Update/UnicodeData-3.0.0.html#General%2=
0Category)
> to identify letters (and now combining marks) and if they are in range,
> performs a substitution. It then uses the transliteration file to find
> rules for particular character substitutions (for example, that file see=
ms
> to handle the copyright symbol substitution). I don't see anything platf=
orm
> dependent in there.

Hm.  Something funny is going on here.  When I fetch the two reference
files from the URLs cited in the script, and do

python2 generate_unaccent_rules.py --unicode-data-file UnicodeData.txt --l=
atin-ascii-file Latin-ASCII.xml >newrules

I get something that's bit-for-bit the same as what's in unaccent.rules.
So there's clearly a platform difference between here and there.

I'm using Python 2.6.6, which is what ships with RHEL6; have not tried
it on anything newer.

			regards, tom lane