Date: Wed, 9 Jan 2019 12:52:53 +0900
From: Michael Paquier <michael@paquier.xyz>
To: Hugh Ranalli <hugh@whtc.ca>
Cc: Tom Lane <tgl@sss.pgh.pa.us>, Alvaro Herrera <alvherre@2ndquadrant.com>,
	Peter Eisentraut <peter.eisentraut@2ndquadrant.com>,
	thomas.munro@enterprisedb.com,
	Daniel Verite <daniel@manitou-mail.org>,
	pgsql-bugs@lists.postgresql.org
Subject: Re: BUG #15548: Unaccent does not remove combining diacritical
 characters
Message-ID: <20190109035253.GF21835@paquier.xyz>
References: <201901031819.hi4hjki2pzr5@alvherre.pgsql>
 <23619.1546539744@sss.pgh.pa.us>
 <CAAhbUMMrNL9uXXpwS+BPTGjKQv5Sxvr+OY2C73Ag7oFyjSOiEQ@mail.gmail.com>
 <20190104130048.GD2067@paquier.xyz>
 <CAAhbUMOj8r4=V0nQpa21SBh7tHTZ37QxXKx5JSpYqdsA-V6mSA@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha512;
	protocol="application/pgp-signature"; boundary="MrRUTeZlqqNo1jQ9"
Content-Disposition: inline
In-Reply-To: 
 <CAAhbUMOj8r4=V0nQpa21SBh7tHTZ37QxXKx5JSpYqdsA-V6mSA@mail.gmail.com>
Precedence: bulk


--MrRUTeZlqqNo1jQ9
Content-Type: multipart/mixed; boundary="sDKAb4OeUBrWWL6P"
Content-Disposition: inline


--sDKAb4OeUBrWWL6P
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline

Hi Hugh,

On Fri, Jan 04, 2019 at 11:29:42AM -0500, Hugh Ranalli wrote:
> I think we're on the same page. I'll wait for you to finish your review and
> provide any further comments before I make any changes.

I have been doing a bit more than a review by studying by myself the
new format and the old format, and the way we could do things in the
XML parsing part, and hacked the code by myself.  On top of the
incorrect URL for Latin-ASCII.xml, I have noticed as well that there
should be only one block transforms/transform/tRule in the source, so
I think that we should add an assertion on that as a sanity check.  I
have also changed the code to use splitlines(), which is more portable
across platforms, and added an extra regression test for the new
characters added to unaccent.rules.  This does not close this thread
but we can support the new format this way.  I have also documented
the way to browse the full set of releases for Latin-ASCII.xml, and
precisely which version has been used for this patch.

This does not close yet the part for diacritical characters, but
supporting the new format is a step into this direction.  What do
you think?
--
Michael

--sDKAb4OeUBrWWL6P
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment; filename="unaccent-format-update.patch"
Content-Transfer-Encoding: quoted-printable

diff --git a/contrib/unaccent/expected/unaccent.out b/contrib/unaccent/expe=
cted/unaccent.out
index 0835e141af..44d70771ac 100644
--- a/contrib/unaccent/expected/unaccent.out
+++ b/contrib/unaccent/expected/unaccent.out
@@ -25,6 +25,12 @@ SELECT unaccent('=D0=81=D0=96=D0=98=D0=9A');
  =D0=95=D0=96=D0=98=D0=9A
 (1 row)
=20
+SELECT unaccent('=CB=83');
+ unaccent=20
+----------
+ >
+(1 row)
+
 SELECT unaccent('unaccent', 'foobar');
  unaccent=20
 ----------
@@ -43,6 +49,12 @@ SELECT unaccent('unaccent', '=D0=81=D0=96=D0=98=D0=9A');
  =D0=95=D0=96=D0=98=D0=9A
 (1 row)
=20
+SELECT unaccent('unaccent', '=CB=83');
+ unaccent=20
+----------
+ >
+(1 row)
+
 SELECT ts_lexize('unaccent', 'foobar');
  ts_lexize=20
 -----------
@@ -61,3 +73,9 @@ SELECT ts_lexize('unaccent', '=D0=81=D0=96=D0=98=D0=9A');
  {=D0=95=D0=96=D0=98=D0=9A}
 (1 row)
=20
+SELECT ts_lexize('unaccent', '=CB=83');
+ ts_lexize=20
+-----------
+ {>}
+(1 row)
+
diff --git a/contrib/unaccent/generate_unaccent_rules.py b/contrib/unaccent=
/generate_unaccent_rules.py
index c9aef490ae..0a181f6857 100644
--- a/contrib/unaccent/generate_unaccent_rules.py
+++ b/contrib/unaccent/generate_unaccent_rules.py
@@ -20,8 +20,13 @@
 # option is enabled, the XML file of this transliterator [2] -- given as a
 # command line argument -- will be parsed and used.
 #
+# Ideally you should use the latest release for each data set.  For
+# Latin-ASCII.xml, the latest data set released can be browsed directly
+# via [3].  Note that this script is compatible with at least release 29.
+#
 # [1] http://unicode.org/Public/8.0.0/ucd/UnicodeData.txt
-# [2] http://unicode.org/cldr/trac/export/12304/tags/release-28/common/tra=
nsforms/Latin-ASCII.xml
+# [2] http://unicode.org/cldr/trac/export/14746/tags/release-34/common/tra=
nsforms/Latin-ASCII.xml
+# [3] https://unicode.org/cldr/trac/browser/tags
=20
 # BEGIN: Python 2/3 compatibility - remove when Python 2 compatibility dro=
pped
 # The approach is to be Python3 compatible with Python2 "backports".
@@ -140,8 +145,18 @@ def parse_cldr_latin_ascii_transliterator(latinAsciiFi=
lePath):
     transliterationTree =3D ET.parse(latinAsciiFilePath)
     transliterationTreeRoot =3D transliterationTree.getroot()
=20
-    for rule in transliterationTreeRoot.findall("./transforms/transform/tR=
ule"):
-        matches =3D rulePattern.search(rule.text)
+    # Fetch all the transformation rules.  Since release 29 of Latin-ASCII=
=2Exml
+    # all the transliteration rules are located in a single tRule block wi=
th
+    # all rules separated into separate lines.
+    blockRules =3D transliterationTreeRoot.findall("./transforms/transform=
/tRule")
+    assert(len(blockRules) =3D=3D 1)
+
+    # Split the block of rules into one element per line.
+    rules =3D blockRules[0].text.splitlines()
+
+    # And finish the processing of each individual rule.
+    for rule in rules:
+        matches =3D rulePattern.search(rule)
=20
         # The regular expression capture four groups corresponding
         # to the characters.
diff --git a/contrib/unaccent/sql/unaccent.sql b/contrib/unaccent/sql/unacc=
ent.sql
index ba72ab6261..d7d3a87e87 100644
--- a/contrib/unaccent/sql/unaccent.sql
+++ b/contrib/unaccent/sql/unaccent.sql
@@ -8,11 +8,14 @@ SET client_encoding TO 'UTF8';
 SELECT unaccent('foobar');
 SELECT unaccent('=D1=91=D0=BB=D0=BA=D0=B0');
 SELECT unaccent('=D0=81=D0=96=D0=98=D0=9A');
+SELECT unaccent('=CB=83');
=20
 SELECT unaccent('unaccent', 'foobar');
 SELECT unaccent('unaccent', '=D1=91=D0=BB=D0=BA=D0=B0');
 SELECT unaccent('unaccent', '=D0=81=D0=96=D0=98=D0=9A');
+SELECT unaccent('unaccent', '=CB=83');
=20
 SELECT ts_lexize('unaccent', 'foobar');
 SELECT ts_lexize('unaccent', '=D1=91=D0=BB=D0=BA=D0=B0');
 SELECT ts_lexize('unaccent', '=D0=81=D0=96=D0=98=D0=9A');
+SELECT ts_lexize('unaccent', '=CB=83');
diff --git a/contrib/unaccent/unaccent.rules b/contrib/unaccent/unaccent.ru=
les
index 76e4e69beb..7ce25eef03 100644
--- a/contrib/unaccent/unaccent.rules
+++ b/contrib/unaccent/unaccent.rules
@@ -399,6 +399,21 @@
 =CA=A6	ts
 =CA=AA	ls
 =CA=AB	lz
+=CA=B9	'
+=CA=BA	"
+=CA=BB	'
+=CA=BC	'
+=CA=BD	'
+=CB=82	<
+=CB=83	>
+=CB=84	^
+=CB=86	^
+=CB=88	'
+=CB=8B	`
+=CB=90	:
+=CB=96	+
+=CB=97	-
+=CB=9C	~
 =CE=86	=CE=91
 =CE=88	=CE=95
 =CE=89	=CE=97

--sDKAb4OeUBrWWL6P--

--MrRUTeZlqqNo1jQ9
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----

iQIzBAABCgAdFiEEG72nH6vTowiyblFKnvQgOdbyQH0FAlw1cBUACgkQnvQgOdby
QH1vRxAAlAxuWdSeFE7+sEemZWlVacPP/HbwTo1f1GjidF4sEXEwy+4DJd2/Skxq
NHuOHx0yCpznFUVnzAs2RDi1jIixn5RJIgROeEN9Lq7J+5O6PQ7FSqOG947aOr1N
toITpeX43MzzOaQpnW2fyc6l0qjxNAuvchWIi9OIRqFvdy503VugtXku4aak4NHM
Kv1wN/M2zhyRrBiPNZzq6BfgnaxYWKvR4plLTvP26LZinGgeHHxLNDfACyesAGgZ
p7SaegxfaCYIP+9tkbVhGEVuXuotM9QUml7w8LbWn6N+LvkeW9KiwVjrI5QIiORK
rC66p9dTn9sdzGv2gPEwVazhp7gmByi/pXQ4uekPYaDPJKEy8cMwGJvwf3l9jEba
5uHlvcj8fbWzS4FChSxCHFCtGvAUqlQBSV9/FaLGDU/JG1Ez+bJNQlKr/xRXGGtN
OWly/pPd4Ok8MV378LCnkHEL6fIvKHbar3hZJ/aaWAV05HuUu89kq9JqTv16Wf71
/QNFrSO/HF50qIgLeGVm0hNSQdLl86PhI0svvvmbkwbn0Bgh8fIA6S6hGiKH/tUR
D6VfYoeSA1LHKWBFNio0ANI+6Krfldc3jE7xP85nVGCMWxkS1ZQiabF1iAShEzN2
WLfeYAhas5r3hs1yaYmzx7YuBqh5PR7jddS8lLJyASdV1uLIctY=
=7Cnt
-----END PGP SIGNATURE-----

--MrRUTeZlqqNo1jQ9--