Message-ID: <660e18263897e046e9104cc60c1cea14dc9e7551.camel@j-davis.com>
Subject: Re: Order changes in PG16 since ICU introduction
From: Jeff Davis <pgsql@j-davis.com>
To: Daniel Verite <daniel@manitou-mail.org>
Cc: Matthias van de Meent <boekewurm+postgres@gmail.com>, Andrew Gierth
 <andrew@tao11.riddles.org.uk>, Peter Eisentraut
 <peter.eisentraut@enterprisedb.com>, Sandro Santilli <strk@kbt.io>, Tom
 Lane <tgl@sss.pgh.pa.us>, Regina Obe <lr@pcorp.us>, 
 pgsql-hackers@lists.postgresql.org
Date: Wed, 24 May 2023 08:39:07 -0700
In-Reply-To: <cb448574-aa7c-4969-b2dd-c9eb221d7e06@manitou-mail.org>
References: <cb448574-aa7c-4969-b2dd-c9eb221d7e06@manitou-mail.org>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
User-Agent: Evolution 3.44.4-0ubuntu1 
MIME-Version: 1.0
Archived-At: <https://www.postgresql.org/message-id/660e18263897e046e9104cc60c1cea14dc9e7551.camel%40j-davis.com>
Precedence: bulk

On Mon, 2023-05-22 at 22:09 +0200, Daniel Verite wrote:
> While I agree that the LOCALE option in CREATE DATABASE is
> counter-intuitive,

I think it's more than that. As Andreww Gierth pointed out:

   $ initdb --locale=3Dfr_FR
   ...
     ICU locale:  en-US
   ...

Is more than just counter-intuitive. I don't think we can ship 16 that
way.

>  I find it questionable that blending ICU
> and libc locales into it helps that much with the user experience.

Thank you for going through some examples here. I agree that it's not
perfect, but we need some path to a reasonable ICU user experience, and
I think we'll have to accept some rough edges to avoid the worst cases,
like above.

> initdb:
>=20
> =C2=A0 Using default ICU locale "fr".
> =C2=A0 Using language tag "fr" for ICU locale "fr".
>=20

...

> #1
>=20
> postgres=3D# create database test1 locale=3D'fr_FR.UTF-8';
> NOTICE:=C2=A0 using standard form "fr-FR" for ICU locale "fr_FR.UTF-8"
> ERROR:=C2=A0=C2=A0new ICU locale (fr-FR) is incompatible with the ICU loc=
ale of

I don't see a problem here. If you specify LOCALE to CREATE DATABASE,
you should either be using "TEMPLATE template0", or you should be
expecting an error if the LOCALE doesn't match exactly.

What would you like to see happen here?

> #2
>=20
> postgres=3D# create database test2 locale=3D'C.UTF-8'
> template=3D'template0';
> NOTICE:=C2=A0 using standard form "en-US-u-va-posix" for ICU locale
> "C.UTF-8"
> CREATE DATABASE
>=20
>=20
> en-US-u-va-posix does not sort like C.UTF-8 in glibc 2.35, so
> this interpretation is arguably not what a user would expect.

As you pointed out, this is not settled in libc either:

https://www.postgresql.org/message-id/8a3dc06f-9b9d-4ed7-9a12-2070d8b0165f%=
40manitou-mail.org

We really can't expect a particular order for a particular locale name,
unless we handle it specially like "C" or "POSIX". If we pass it to the
provider, we have to trust the provider to match our conceptual
expectations for that locale (and ideally version it properly).

> I would expect the ICU warning or error (icu_validation_level) to
> kick
> in instead of that transliteration.

Earlier versions of ICU (<=3D 63) do this transformation automatically,
and I don't see a reason to throw an error if ICU considers it valid.
The language tag en-US-u-va-posix will be stored in the catalog, and
that will be considered valid in later versions of ICU.

Later versions of ICU (>=3D 64) consider locales with a language name of
"C" to be obsolete and no longer valid. I added code to do the
transformation without error in these later versions, but I think we
have agreement to remove it.

If a user specifies the locale as "C.UTF-8", we can either pass it to
ICU and see whether that version accepts it or not (and if not, throw a
warning/error); or if we decide that "C.UTF-8" really means "C", we can
handle it in the memcmp() path like C and never send it to ICU.

> #3
>=20
> $ grep french /etc/locale.alias
> french=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0fr_FR.I=
SO-8859-1
>=20
> postgres=3D# create database test3 locale=3D'french' template=3D'template=
0'
> encoding=3D'LATIN1';
> WARNING:=C2=A0 ICU locale "french" has unknown language "french"
> HINT:=C2=A0 To disable ICU locale validation, set parameter
> icu_validation_level
> to DISABLED.
> CREATE DATABASE
>=20
>=20
> In practice we're probably getting the "und" ICU locale whereas "fr"
> would
> be appropriate.

This is a good point and illustrates that ICU is not a drop-in
replacement for libc in all cases.

I don't see a solution here that doesn't involve some rough edges,
though. "Locale" is a generic term, and if we continue to insist that
it really means a libc locale, then ICU will never be on an equal
footing with libc, let alone the preferred provider.

Regards,
	Jeff Davis