Date: Tue, 18 Dec 2018 13:57:08 +0900
From: Michael Paquier <michael@paquier.xyz>
To: Thomas Munro <thomas.munro@enterprisedb.com>
Cc: hugh@whtc.ca, Tom Lane <tgl@sss.pgh.pa.us>, daniel@manitou-mail.org,
	PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
Subject: Re: BUG #15548: Unaccent does not remove combining diacritical
 characters
Message-ID: <20181218045708.GI1532@paquier.xyz>
References: 
 <CAAhbUMN1n=ZVns-OeCbaVRYPS0oj7tTnmJrzw7Az-op4DHC+JA@mail.gmail.com>
 <23237.1544899488@sss.pgh.pa.us>
 <CAAhbUMO+fTZTwigDJ=tB3qFvHMe-xAJO5QpsFPH6Vb2oDYAU6w@mail.gmail.com>
 <CAAhbUMMmXnj0YSD+fr5hSqeC+D6PAG+0kXJwMMhK2DCdwQVoxQ@mail.gmail.com>
 <29419.1544908811@sss.pgh.pa.us>
 <CAEepm=3gSmNWkteBxCEL-W+j1dmbcNzDin_iv+f_Om6o+1fAiA@mail.gmail.com>
 <CAAhbUMOX4QLj6c0O3GnjZYtR2dpAowss832Bq1n7oJyByeR7kQ@mail.gmail.com>
 <31564.1545078667@sss.pgh.pa.us>
 <CAAhbUMOe_VeJBxVqPe4Op2XKrT+xCcCvsUYRH4v6G71NDGG=fg@mail.gmail.com>
 <CAEepm=0qb_nx-f8cACS1=1NdmCj-3D9zXFU+RJHsFbZEztcqjg@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha512;
	protocol="application/pgp-signature"; boundary="KrHCbChajFcK0yQE"
Content-Disposition: inline
In-Reply-To: 
 <CAEepm=0qb_nx-f8cACS1=1NdmCj-3D9zXFU+RJHsFbZEztcqjg@mail.gmail.com>
Precedence: bulk


--KrHCbChajFcK0yQE
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On Tue, Dec 18, 2018 at 03:05:00PM +1100, Thomas Munro wrote:
> I don't think this is quite right.  Those don't seem to be the
> combining codepoints[1], and in any case they are being replaced with
> ASCII characters, whereas I thought we wanted to replace them with
> nothing at all.  Here is my attempt to come up with a test case using
> combining characters:
>=20
>   select unaccent('un cafe=CC=81 cre=CC=80me s''il vous plai=CC=82t');
>=20
> It's not stripping the accents.  I've attached that in a file for
> reference so you can run it with psql -f x.sql, and you can see that
> it's using combining code points (code points 0301, 0300, 0302 which
> come out as cc81, cc80, cc82 in UTF-8) like so:

Could you also add some tests in contrib/unaccent/sql/unaccent.sql at
the same time?  That would be nice to check easily the extent of the
patches proposed on this thread.
--
Michael

--KrHCbChajFcK0yQE
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----

iQIzBAABCgAdFiEEG72nH6vTowiyblFKnvQgOdbyQH0FAlwYfiQACgkQnvQgOdby
QH1QCg/+MlB6AjEQA4BLoplsbhjD13Bn24N+rle+x5e1oJYLxNzqTxeI5/0dCAGL
sa4Xqr6vieJhxaRn7n7CcR5qpzweGys9DBdta80W9F5swmXC+A/QB9s0RjrRpZv0
DO8t7bDh90t4/ia+Y1U8K9E106cVclMOt9KJihRUoBEXXxRttd+/dTqDWlhyu7ol
PbzUVIxkYJstOrSaRnSb8dNkGFGz7P2Ix2l6M4rscJj21PDDoLkrhAe77RKLRBoP
kDD05eYw5WxSMTA1ikLZwNztnem8Xog46t1MUq8AijsmlcWI/AcXg5SzLklBEe6Q
Xya7sPRdoE20pcA/RM9UUUoPvrhmfc0eZe37lPmD528Ts/4V/NqwjMUQYSXHB3eg
Fg/DaGGy8YvNDKZ6GK15cbBPWP/qBAVDb7MxwM/BCyOx0tAfm3Xmj6IDArJ1qti+
EcMj+T9s9JuHlceMy5ZQZBz2WAY7giB/cg2F1g76Cf6WBZK5TDB+H5TQ1Gvnxa3i
PuQygKrkBGtjss7n38/Nwe6WlsD458J9nNU/oC4Jfdn/B+3DdnPRFhPiiOqwKJRp
okA6psk1wsVqRnWr+0y7CzjrGY83G7RI51r+J2HvSt07EGdxEBmxwP9Y3rwKanxF
QXLIoRJHdeRzux0txLbB/hv3RQV/DlEuErnMoUxrf5wSF3QCwKY=
=uA4B
-----END PGP SIGNATURE-----

--KrHCbChajFcK0yQE--