pg.ddx.io  pgsql-bugs@postgresql.org mailing list archive  
help / color / mirror / Atom feed
From: PG Bug reporting form <noreply@postgresql.org>
To: pgsql-bugs@lists.postgresql.org
Cc: hugh@whtc.ca
Subject: BUG #15548: Unaccent does not remove combining diacritical characters
Date: Wed, 12 Dec 2018 20:00:45 +0000
Message-ID: <15548-cef1b3f8de190d4f@postgresql.org> (raw)

The following bug has been logged on the website:

Bug reference:      15548
Logged by:          Hugh Ranalli
Email address:      hugh@whtc.ca
PostgreSQL version: 11.1
Operating system:   Ubuntu 18.04
Description:        

Apparently Unicode has two ways of accenting a character: as a separate code
point, which represents the base character and the accent, or as a
"combining diacritical mark"
(https://en.wikipedia.org/wiki/Combining_Diacritical_Marks), in which case
the mark applies itself to the preceding character. For example, A followed
by U+0300 displays À. However, unaccent is not removing these accents. 

SELECT unaccent(U&'A\0300'); should result in 'A', but instead results in
'À.' I'm running PostgreSQL 11.1, installed from the PostgreSQL
repositories. I've read bug report #13440, and have tried with both the
installed unaccent.rules as well as a new set generated by the
generate_unaccent_rules.py distributed with the 11.1 source code:
  wget http://unicode.org/Public/7.0.0/ucd/UnicodeData.txt
  wget
https://www.unicode.org/repos/cldr/trunk/common/transforms/Latin-ASCII.xml
  python generate_unaccent_rules.py --unicode-data-file UnicodeData.txt
--latin-ascii-file  Latin-ASCII.xml > unaccent.rules

I see there have been some updates to generate_unaccent_rules.py to handle
Greek and Vietnamese characters, but neither of them seem to address this
issue. I'm happy to contribute a patch to handle these cases, but of course
wanted to make sure this is desired behaviour, or if I am misunderstanding
something somewhere.

Thank you,
Hugh Ranalli



view thread (59+ messages)  latest in thread

Message-ID: <15548-cef1b3f8de190d4f@postgresql.org>
Permalink:  ../15548-cef1b3f8de190d4f@postgresql.org/
Also on:    postgresql.org/message-id/15548-cef1b3f8de190d4f@postgresql.org

 · 

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients using the --to and --cc options:
  reply via email

  To: pgsql-bugs@postgresql.org
  Cc: noreply@postgresql.org, pgsql-bugs@lists.postgresql.org, hugh@whtc.ca
  Subject: Re: BUG #15548: Unaccent does not remove combining diacritical characters
  In-Reply-To: <15548-cef1b3f8de190d4f@postgresql.org>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox