From: Tom Lane <tgl@sss.pgh.pa.us>
To: Michael Paquier <michael@paquier.xyz>
cc: Thomas Munro <thomas.munro@enterprisedb.com>, hugh@whtc.ca,
        daniel@manitou-mail.org,
        PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
Subject: Re: BUG #15548: Unaccent does not remove combining diacritical
 characters
In-reply-to: <20181218045708.GI1532@paquier.xyz>
References: 
 <CAAhbUMN1n=ZVns-OeCbaVRYPS0oj7tTnmJrzw7Az-op4DHC+JA@mail.gmail.com>
 <23237.1544899488@sss.pgh.pa.us>
 <CAAhbUMO+fTZTwigDJ=tB3qFvHMe-xAJO5QpsFPH6Vb2oDYAU6w@mail.gmail.com>
 <CAAhbUMMmXnj0YSD+fr5hSqeC+D6PAG+0kXJwMMhK2DCdwQVoxQ@mail.gmail.com>
 <29419.1544908811@sss.pgh.pa.us>
 <CAEepm=3gSmNWkteBxCEL-W+j1dmbcNzDin_iv+f_Om6o+1fAiA@mail.gmail.com>
 <CAAhbUMOX4QLj6c0O3GnjZYtR2dpAowss832Bq1n7oJyByeR7kQ@mail.gmail.com>
 <31564.1545078667@sss.pgh.pa.us>
 <CAAhbUMOe_VeJBxVqPe4Op2XKrT+xCcCvsUYRH4v6G71NDGG=fg@mail.gmail.com>
 <CAEepm=0qb_nx-f8cACS1=1NdmCj-3D9zXFU+RJHsFbZEztcqjg@mail.gmail.com>
 <20181218045708.GI1532@paquier.xyz>
Comments: In-reply-to Michael Paquier <michael@paquier.xyz>
	message dated "Tue, 18 Dec 2018 13:57:08 +0900"
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-ID: <8505.1545111362.1@sss.pgh.pa.us>
Date: Tue, 18 Dec 2018 00:36:02 -0500
Message-ID: <8506.1545111362@sss.pgh.pa.us>
Precedence: bulk

Michael Paquier <michael@paquier.xyz> writes:
> Could you also add some tests in contrib/unaccent/sql/unaccent.sql at
> the same time?  That would be nice to check easily the extent of the
> patches proposed on this thread.

I wonder why unaccent.sql is set up to run its tests in KOI8 client
encoding rather than UTF8.  It doesn't seem like it's the business
of this test script to be verifying transcoding from KOI8 to UTF8
(and if it were meant to do that, it's a pretty incomplete test...).
But having it set up like that means that we can't directly add
such tests to unaccent.sql, because there are no combining diacritics
in the KOI8 character set.  We have two unattractive options:

* Change client encodings partway through unaccent.sql.  I think this
would be disastrous for editability of that file; no common tools
will understand the encoding change.

* Put the new test cases into a separate file with a different client
encoding.  This is workable, I suppose, but it seems pretty silly
when the tests are only a few queries apiece.

Another problem I've got with the current setup is that it seems
unlikely that many people's editors default to an assumption of
KOI8 encoding.  Mine guesses that these files are UTF8, and so
the test cases look perfectly insane.  They do make sense if
I transcode the files to UTF8, but I wonder why we're not shipping
them as UTF8 in the first place.

tl;dr: I think we should convert unaccent.sql and unaccent.out
to UTF8 encoding.  Then, adding more test cases for this patch
will be easy.

			regards, tom lane