Content-Type: text/plain; charset="utf-8"
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
From: "Daniel Verite" <daniel@manitou-mail.org>
Subject: Re: Built-in CTYPE provider
To: "Jeff Davis" <pgsql@j-davis.com>
Cc: Peter Eisentraut <peter@eisentraut.org>,
 Robert Haas <robertmhaas@gmail.com>,
 Jeremy Schneider <schneider@ardentperf.com>,
 pgsql-hackers@postgresql.org
In-Reply-To: <a21b515000f121244db2a5570d4292d19df23379.camel@j-davis.com>
Date: Wed, 27 Mar 2024 16:53:33 +0100
Message-Id: <610d7f1b-c68c-4eb8-a03d-1515da304c58@manitou-mail.org>
Archived-At: <https://www.postgresql.org/message-id/610d7f1b-c68c-4eb8-a03d-1515da304c58%40manitou-mail.org>
Precedence: bulk

	Jeff Davis wrote:

> The tests include initcap('123abc') which is '123abc' in the PG_C_UTF8
> collation vs '123Abc' in PG_UNICODE_FAST.
>=20
> The reason for the latter behavior is that the Unicode Default Case
> Conversion algorithm for toTitlecase() advances to the next Cased
> character before mapping to titlecase, and digits are not Cased. ICU
> has a configurable adjustment, and defaults in a way that produces
> '123abc'.

Even aside from ICU, there's a different behavior between glibc
and pg_c_utf8 glibc for codepoints in the decimal digit category=20
outside of the US-ASCII range '0'..'9',

select initcap(concat(chr(0xff11), 'a') collate "C.utf8");   -- glibc 2.35
 initcap=20
---------
 =EF=BC=91a

select initcap(concat(chr(0xff11), 'a') collate "pg_c_utf8");
 initcap=20
---------
 =EF=BC=91A

Both collations consider that chr(0xff11) is not a digit
(isdigit()=3D>false) but C.utf8 says that it's alpha, whereas pg_c_utf8
says it's neither digit nor alpha.

AFAIU this is why in the above initcap() call, pg_c_utf8 considers
that 'a' is the first alphanumeric, whereas C.utf8 considers that '=EF=BC=
=91'
is the first alphanumeric, leading to different capitalizations.

Comparing the 3 providers:

WITH v(provider,type,result) AS (values
 ('ICU', 'isalpha', chr(0xff11) ~ '[[:alpha:]]' collate "unicode"),
 ('glibc', 'isalpha', chr(0xff11) ~ '[[:alpha:]]' collate "C.utf8"),
 ('builtin', 'isalpha', chr(0xff11) ~ '[[:alpha:]]' collate "pg_c_utf8"),
 ('ICU', 'isdigit', chr(0xff11) ~ '[[:digit:]]' collate "unicode"),
 ('glibc', 'isdigit', chr(0xff11) ~ '[[:digit:]]' collate "C.utf8"),
 ('builtin', 'isdigit', chr(0xff11) ~ '[[:digit:]]' collate "pg_c_utf8")
 )
select * from v
\crosstabview


 provider | isalpha | isdigit=20
----------+---------+---------
 ICU	  | f	    | t
 glibc	  | t	    | f
 builtin  | f	    | f


Are we fine with pg_c_utf8 differing from both ICU's point of view
(U+ff11 is digit and not alpha) and glibc point of view (U+ff11 is not
digit, but it's alpha)?

Aside from initcap(), this is going to be significant for regular
expressions.


Best regards,
--=20
Daniel V=C3=A9rit=C3=A9
https://postgresql.verite.pro/
Twitter: @DanielVerite