Message-ID: <d321c19850a236e76850f00437a34b8f8f1abb90.camel@j-davis.com>
Subject: Re: Order changes in PG16 since ICU introduction
From: Jeff Davis <pgsql@j-davis.com>
To: Tom Lane <tgl@sss.pgh.pa.us>
Cc: Regina Obe <lr@pcorp.us>, 'Peter Eisentraut'
	 <peter.eisentraut@enterprisedb.com>, 'Sandro Santilli' <strk@kbt.io>, 
	pgsql-hackers@lists.postgresql.org
Date: Mon, 24 Apr 2023 21:31:40 -0700
In-Reply-To: <3391932.1682107209@sss.pgh.pa.us>
References: <000b01d97465$c34bbd60$49e33820$@pcorp.us>
	 <3353947.1682092131@sss.pgh.pa.us> <ZELDOktA75yYG4Z2@hst.kbt.io>
	 <533d93c5-f604-8e5f-4a48-975c08c53d59@enterprisedb.com>
	 <3365333.1682098091@sss.pgh.pa.us>
	 <002d01d97477$f0395640$d0ac02c0$@pcorp.us>
	 <3367226.1682099185@sss.pgh.pa.us>
	 <003b01d9747a$910471c0$b30d5540$@pcorp.us>
	 <3368986.1682099982@sss.pgh.pa.us>
	 <000001d9747d$066b7560$13426020$@pcorp.us>
	 <3372735.1682101425@sss.pgh.pa.us>
	 <f83f089ee1e9acd5dbbbf3353294d24e1f196e95.camel@j-davis.com>
	 <3391932.1682107209@sss.pgh.pa.us>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
User-Agent: Evolution 3.44.4-0ubuntu1 
MIME-Version: 1.0
Archived-At: <https://www.postgresql.org/message-id/d321c19850a236e76850f00437a34b8f8f1abb90.camel%40j-davis.com>
Precedence: bulk

On Fri, 2023-04-21 at 16:00 -0400, Tom Lane wrote:
> I think I might like this idea, except for one thing: you're
> imagining
> that the locale doesn't control anything except string comparisons.
> What about to_upper/to_lower, character classifications in regexes,
> etc?

If provider=3D'libc' and LC_CTYPE=3D'C', str_toupper/str_tolower are
handled with asc_tolower/asc_toupper. The regex character
classification is done with pg_char_properties. In these cases neither
ICU nor libc is used; it's just code in postgres.

libc is special in that you can set LC_COLLATE and LC_CTYPE separately,
so that different locales are used for sorting and character
classification. That's potentially useful to set LC_COLLATE to C for
performance reasons, while setting LC_CTYPE to a useful locale. We
don't allow ICU to set collation and ctype separately (it would be
possible to allow it, but I don't think there's a huge demand and it's
arguably inconsistent to set them differently).

> (I'm not sure whether those operations can get redirected to ICU
> today
> or whether they still always go to libc, but we'll surely want to fix
> it eventually if the latter is still true.)

Those operations do get redirected to ICU today. There are extensions
that call locale-sensitive libc functions directly, and obviously those
won't use ICU.


> Aside from the user-surprise issues discussed up to now, pg_dump
> scripts
> emitted by pre-v15 pg_dump are not going to contain LOCALE_PROVIDER
> clauses in CREATE DATABASE, and people are going to be very unhappy
> if that means they suddenly get totally different locale semantics
> after restoring into a new DB.

Agreed.

> =C2=A0 I think we need some plan for mapping
> libc-style locale specs into ICU locales so that we can make that
> more nearly transparent.

ICU does a reasonable job mapping libc-like locale names to ICU
locales, e.g. en_US to en-US, etc. The ordering semantics aren't
guaranteed to be the same, of course (because the libc-locales are
platform-dependent), but it's at least conceptually the same locale.

>=20
> Maybe this means we are not ready to do ICU-by-default in v16.
> It certainly feels like there might be more here than we want to
> start designing post-feature-freeze.

This thread is already on the Open Items list. As long as it's not too
disruptive to others I'll leave it as-is for now to see how this sorts
out. Right now it's not clear to me how much of this is a v15 issue vs
a v16 issue.

Regards,
	Jeff Davis