Re: BUG #15548: Unaccent does not remove combining diacritical characters

pg.ddx.io  pgsql-bugs@postgresql.org mailing list archive  
help / color / mirror / Atom feed

From: Michael Paquier <michael@paquier.xyz>
To: Thomas Munro <thomas.munro@enterprisedb.com>
Cc: hugh@whtc.ca, Tom Lane <tgl@sss.pgh.pa.us>
Cc: daniel@manitou-mail.org, PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
Subject: Re: BUG #15548: Unaccent does not remove combining diacritical characters
Date: Tue, 18 Dec 2018 13:57:08 +0900
Message-ID: <20181218045708.GI1532@paquier.xyz> (raw)
In-Reply-To: <CAEepm=0qb_nx-f8cACS1=1NdmCj-3D9zXFU+RJHsFbZEztcqjg@mail.gmail.com>
References: <CAAhbUMN1n=ZVns-OeCbaVRYPS0oj7tTnmJrzw7Az-op4DHC+JA@mail.gmail.com>
	<23237.1544899488@sss.pgh.pa.us>
	<CAAhbUMO+fTZTwigDJ=tB3qFvHMe-xAJO5QpsFPH6Vb2oDYAU6w@mail.gmail.com>
	<CAAhbUMMmXnj0YSD+fr5hSqeC+D6PAG+0kXJwMMhK2DCdwQVoxQ@mail.gmail.com>
	<29419.1544908811@sss.pgh.pa.us>
	<CAEepm=3gSmNWkteBxCEL-W+j1dmbcNzDin_iv+f_Om6o+1fAiA@mail.gmail.com>
	<CAAhbUMOX4QLj6c0O3GnjZYtR2dpAowss832Bq1n7oJyByeR7kQ@mail.gmail.com>
	<31564.1545078667@sss.pgh.pa.us>
	<CAAhbUMOe_VeJBxVqPe4Op2XKrT+xCcCvsUYRH4v6G71NDGG=fg@mail.gmail.com>
	<CAEepm=0qb_nx-f8cACS1=1NdmCj-3D9zXFU+RJHsFbZEztcqjg@mail.gmail.com>

On Tue, Dec 18, 2018 at 03:05:00PM +1100, Thomas Munro wrote:
> I don't think this is quite right.  Those don't seem to be the
> combining codepoints[1], and in any case they are being replaced with
> ASCII characters, whereas I thought we wanted to replace them with
> nothing at all.  Here is my attempt to come up with a test case using
> combining characters:
> 
>   select unaccent('un café crème s''il vous plaît');
> 
> It's not stripping the accents.  I've attached that in a file for
> reference so you can run it with psql -f x.sql, and you can see that
> it's using combining code points (code points 0301, 0300, 0302 which
> come out as cc81, cc80, cc82 in UTF-8) like so:

Could you also add some tests in contrib/unaccent/sql/unaccent.sql at
the same time?  That would be nice to check easily the extent of the
patches proposed on this thread.
--
Michael

Attachments:

  [application/pgp-signature] signature.asc (832B, ../20181218045708.GI1532@paquier.xyz/2-signature.asc)
  download

view thread (59+ messages)  latest in thread

Message-ID: <20181218045708.GI1532@paquier.xyz>
Permalink:  ../20181218045708.GI1532@paquier.xyz/
Also on:    postgresql.org/message-id/20181218045708.GI1532@paquier.xyz

 ·

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients using the --to and --cc options:
  reply via email

  To: pgsql-bugs@postgresql.org
  Cc: michael@paquier.xyz, thomas.munro@enterprisedb.com, tgl@sss.pgh.pa.us, pgsql-bugs@lists.postgresql.org
  Subject: Re: BUG #15548: Unaccent does not remove combining diacritical characters
  In-Reply-To: <20181218045708.GI1532@paquier.xyz>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox