From: Dag Lem <dag@nimrod.no>
To: Tom Lane <tgl@sss.pgh.pa.us>
Cc: Thomas Munro <thomas.munro@gmail.com>, Andres Freund <andres@anarazel.de>,
        PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Subject: Re: daitch_mokotoff module
Organization: Nimrod
References: <yger1atbgfy.fsf@sid.nimrod.no> <ygek0g87hgx.fsf@sid.nimrod.no>
	<0cac2d60-5b85-59c6-1ac1-77092b8688b7@enterprisedb.com>
	<ygelf0mizon.fsf@sid.nimrod.no> <yge7dbxhc01.fsf@sid.nimrod.no>
	<20220102213150.y2tpuwqhruu5t2y5@alap3.anarazel.de>
	<CA+hUKG+Gs3WXzbT4OK9gEXpC1e221RafrPS4LVvcDDgDi20xAg@mail.gmail.com>
	<3461033.1641177713@sss.pgh.pa.us>
Date: Mon, 03 Jan 2022 14:07:09 +0100
In-Reply-To: <3461033.1641177713@sss.pgh.pa.us> (Tom Lane's message of "Sun,
	02 Jan 2022 21:41:53 -0500")
Message-ID: <ygeo84tvugy.fsf@sid.nimrod.no>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.3 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
Archived-At: <https://www.postgresql.org/message-id/ygeo84tvugy.fsf%40sid.nimrod.no>
Precedence: bulk

Tom Lane <tgl@sss.pgh.pa.us> writes:

> Thomas Munro <thomas.munro@gmail.com> writes:
>> Erm, it looks like something weird is happening somewhere in cfbot's
>> pipeline, because Dag's patch says:
>
>> +SELECT daitch_mokotoff('Stra=C3=9Fburg');
>> + daitch_mokotoff
>> +-----------------
>> + 294795
>> +(1 row)
>
> ... so, that test case is guaranteed to fail in non-UTF8 encodings,
> I suppose?  I wonder what the LANG environment is in that cfbot
> instance.
>
> (We do have methods for dealing with non-ASCII test cases, but
> I can't see that this patch is using any of them.)
>
> 			regards, tom lane
>

I naively assumed that tests would be run in an UTF8 environment.

Running "ack -l '[\x80-\xff]'" in the contrib/ directory reveals that
two other modules are using UTF8 characters in tests - citext and
unaccent.

The citext tests seem to be commented out - "Multibyte sanity
tests. Uncomment to run."

Looking into the unaccent module, I don't quite understand how it will
work with various encodings, since it doesn't seem to decode its input -
will it fail if run under anything but ASCII or UTF8?

In any case, I see that unaccent.sql starts as follows:


CREATE EXTENSION unaccent;

-- must have a UTF8 database
SELECT getdatabaseencoding();

SET client_encoding TO 'UTF8';


Would doing the same thing in fuzzystrmatch.sql fix the problem with
failing tests? Should I prepare a new patch?


Best regards

Dag Lem