From: Michael Paquier <michael@paquier.xyz>
To: Hugh Ranalli <hugh@whtc.ca>
Cc: Tom Lane <tgl@sss.pgh.pa.us>
Cc: Alvaro Herrera <alvherre@2ndquadrant.com>
Cc: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
Cc: thomas.munro@enterprisedb.com, Daniel Verite <daniel@manitou-mail.org>
Cc: pgsql-bugs@lists.postgresql.org
Subject: Re: BUG #15548: Unaccent does not remove combining diacritical characters
Date: Wed, 9 Jan 2019 12:52:53 +0900
Message-ID: <20190109035253.GF21835@paquier.xyz> (raw)
In-Reply-To: <CAAhbUMOj8r4=V0nQpa21SBh7tHTZ37QxXKx5JSpYqdsA-V6mSA@mail.gmail.com>
References: <201901031819.hi4hjki2pzr5@alvherre.pgsql>
<23619.1546539744@sss.pgh.pa.us>
<CAAhbUMMrNL9uXXpwS+BPTGjKQv5Sxvr+OY2C73Ag7oFyjSOiEQ@mail.gmail.com>
<20190104130048.GD2067@paquier.xyz>
<CAAhbUMOj8r4=V0nQpa21SBh7tHTZ37QxXKx5JSpYqdsA-V6mSA@mail.gmail.com>
Hi Hugh,
On Fri, Jan 04, 2019 at 11:29:42AM -0500, Hugh Ranalli wrote:
> I think we're on the same page. I'll wait for you to finish your review and
> provide any further comments before I make any changes.
I have been doing a bit more than a review by studying by myself the
new format and the old format, and the way we could do things in the
XML parsing part, and hacked the code by myself. On top of the
incorrect URL for Latin-ASCII.xml, I have noticed as well that there
should be only one block transforms/transform/tRule in the source, so
I think that we should add an assertion on that as a sanity check. I
have also changed the code to use splitlines(), which is more portable
across platforms, and added an extra regression test for the new
characters added to unaccent.rules. This does not close this thread
but we can support the new format this way. I have also documented
the way to browse the full set of releases for Latin-ASCII.xml, and
precisely which version has been used for this patch.
This does not close yet the part for diacritical characters, but
supporting the new format is a step into this direction. What do
you think?
--
Michael
Attachments:
[text/x-diff] unaccent-format-update.patch (3.8K, ../20190109035253.GF21835@paquier.xyz/2-unaccent-format-update.patch)
download | inline diff:diff --git a/contrib/unaccent/expected/unaccent.out b/contrib/unaccent/expected/unaccent.outindex 0835e141af..44d70771ac 100644--- a/contrib/unaccent/expected/unaccent.out+++ b/contrib/unaccent/expected/unaccent.out@@ -25,6 +25,12 @@ SELECT unaccent('ЁЖИК');
ЕЖИК
(1 row)
+SELECT unaccent('˃');+ unaccent +----------+ >+(1 row)+
SELECT unaccent('unaccent', 'foobar');
unaccent
----------
@@ -43,6 +49,12 @@ SELECT unaccent('unaccent', 'ЁЖИК');
ЕЖИК
(1 row)
+SELECT unaccent('unaccent', '˃');+ unaccent +----------+ >+(1 row)+
SELECT ts_lexize('unaccent', 'foobar');
ts_lexize
-----------
@@ -61,3 +73,9 @@ SELECT ts_lexize('unaccent', 'ЁЖИК');
{ЕЖИК}
(1 row)
+SELECT ts_lexize('unaccent', '˃');+ ts_lexize +-----------+ {>}+(1 row)+diff --git a/contrib/unaccent/generate_unaccent_rules.py b/contrib/unaccent/generate_unaccent_rules.pyindex c9aef490ae..0a181f6857 100644--- a/contrib/unaccent/generate_unaccent_rules.py+++ b/contrib/unaccent/generate_unaccent_rules.py@@ -20,8 +20,13 @@
# option is enabled, the XML file of this transliterator [2] -- given as a
# command line argument -- will be parsed and used.
#
+# Ideally you should use the latest release for each data set. For+# Latin-ASCII.xml, the latest data set released can be browsed directly+# via [3]. Note that this script is compatible with at least release 29.+#
# [1] http://unicode.org/Public/8.0.0/ucd/UnicodeData.txt
-# [2] http://unicode.org/cldr/trac/export/12304/tags/release-28/common/transforms/Latin-ASCII.xml+# [2] http://unicode.org/cldr/trac/export/14746/tags/release-34/common/transforms/Latin-ASCII.xml+# [3] https://unicode.org/cldr/trac/browser/tags
# BEGIN: Python 2/3 compatibility - remove when Python 2 compatibility dropped
# The approach is to be Python3 compatible with Python2 "backports".
@@ -140,8 +145,18 @@ def parse_cldr_latin_ascii_transliterator(latinAsciiFilePath):
transliterationTree = ET.parse(latinAsciiFilePath)
transliterationTreeRoot = transliterationTree.getroot()
- for rule in transliterationTreeRoot.findall("./transforms/transform/tRule"):- matches = rulePattern.search(rule.text)+ # Fetch all the transformation rules. Since release 29 of Latin-ASCII.xml+ # all the transliteration rules are located in a single tRule block with+ # all rules separated into separate lines.+ blockRules = transliterationTreeRoot.findall("./transforms/transform/tRule")+ assert(len(blockRules) == 1)++ # Split the block of rules into one element per line.+ rules = blockRules[0].text.splitlines()++ # And finish the processing of each individual rule.+ for rule in rules:+ matches = rulePattern.search(rule)
# The regular expression capture four groups corresponding
# to the characters.
diff --git a/contrib/unaccent/sql/unaccent.sql b/contrib/unaccent/sql/unaccent.sqlindex ba72ab6261..d7d3a87e87 100644--- a/contrib/unaccent/sql/unaccent.sql+++ b/contrib/unaccent/sql/unaccent.sql@@ -8,11 +8,14 @@ SET client_encoding TO 'UTF8';
SELECT unaccent('foobar');
SELECT unaccent('ёлка');
SELECT unaccent('ЁЖИК');
+SELECT unaccent('˃');
SELECT unaccent('unaccent', 'foobar');
SELECT unaccent('unaccent', 'ёлка');
SELECT unaccent('unaccent', 'ЁЖИК');
+SELECT unaccent('unaccent', '˃');
SELECT ts_lexize('unaccent', 'foobar');
SELECT ts_lexize('unaccent', 'ёлка');
SELECT ts_lexize('unaccent', 'ЁЖИК');
+SELECT ts_lexize('unaccent', '˃');diff --git a/contrib/unaccent/unaccent.rules b/contrib/unaccent/unaccent.rulesindex 76e4e69beb..7ce25eef03 100644--- a/contrib/unaccent/unaccent.rules+++ b/contrib/unaccent/unaccent.rules@@ -399,6 +399,21 @@
ʦ ts
ʪ ls
ʫ lz
+ʹ '+ʺ "+ʻ '+ʼ '+ʽ '+˂ <+˃ >+˄ ^+ˆ ^+ˈ '+ˋ `+ː :+˖ ++˗ -+˜ ~
Ά Α
Έ Ε
Ή Η
[application/pgp-signature] signature.asc (832B, ../20190109035253.GF21835@paquier.xyz/3-signature.asc)
download
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-bugs@postgresql.org
Cc: michael@paquier.xyz, hugh@whtc.ca, tgl@sss.pgh.pa.us, alvherre@2ndquadrant.com, peter.eisentraut@2ndquadrant.com, daniel@manitou-mail.org, pgsql-bugs@lists.postgresql.org
Subject: Re: BUG #15548: Unaccent does not remove combining diacritical characters
In-Reply-To: <20190109035253.GF21835@paquier.xyz>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox