Message-ID: <65e01c2d170a2f80cc8d2bcd144acaf37d31230c.camel@j-davis.com>
Subject: Re: Built-in CTYPE provider
From: Jeff Davis <pgsql@j-davis.com>
To: Jeremy Schneider <schneider@ardentperf.com>, Daniel Verite
	 <daniel@manitou-mail.org>
Cc: Robert Haas <robertmhaas@gmail.com>, pgsql-hackers@postgresql.org
Date: Tue, 09 Jan 2024 14:55:44 -0800
In-Reply-To: <63cd8625-68fa-4760-844a-6b7f643336f2@ardentperf.com>
References: <dd9261f4-7a98-4565-93ec-336c1c110d90@manitou-mail.org>
	 <6b1370d5eaba5e8c42f54c05f7bc2b8e27b8db12.camel@j-davis.com>
	 <804eb67b37f41d3afeb2b6469cbe8bfa79c562cc.camel@j-davis.com>
	 <63cd8625-68fa-4760-844a-6b7f643336f2@ardentperf.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
User-Agent: Evolution 3.44.4-0ubuntu2 
MIME-Version: 1.0
Archived-At: 
 <https://www.postgresql.org/message-id/65e01c2d170a2f80cc8d2bcd144acaf37d31230c.camel%40j-davis.com>
Precedence: bulk

On Mon, 2024-01-08 at 17:17 -0800, Jeremy Schneider wrote:

> I agree with merging the threads, even though it makes for a larger
> patch set. It would be great to get a unified "builtin" provider in
> place for the next major.

I believe that's possible and that this proposal is quite close (hoping
to get something in this 'fest). The tables I'm introducing have
exhaustive test coverage, so there's not a lot of risk there. And the
builtin provider itself is an optional feature, so it won't be
disruptive.

>=20
> In the first list it seems that some callers might be influenced by a
> COLLATE clause or table definition while others always take the
> database
> default? It still seems a bit odd to me if different providers can be
> used for different parts of a single SQL.

Right, that can happen today, and my proposal doesn't change that.
Basically those are cases where the caller was never properly onboarded
to our collation system, like the ts_locale.c routines.

> Is there any reason we couldn't commit the minor cleanup (patch 0001)
> now? It's less than 200 lines and pretty straightforward.

Sure, I'll commit that fairly soon then.

> I wonder if, after a year of running the builtin provider in
> production,
> whether we might consider adding to the builtin provider a few
> locales
> with simple but more reasonable ordering for european and asian
> languages?

I won't rule that out completely, but there's a lot we would need to do
to get there. Even assuming we implement that perfectly, we'd need to
make sure it's a reasonable scope for Postgres as a project and that we
have more than one person willing to maintain it. Similar things have
been rejected before for similar reasons.

What I'm proposing for v17 is much simpler: basically some lookup
tables, which is just an extension of what we're already doing for
normalization.

> https://jeremyhussell.blogspot.com/2017/11/falsehoods-programmers-believe=
-about.html#main
>=20
> Make sure to click the link to show the counterexamples and
> discussion,
> that's the best part.

Yes, it can be hard to reason about this stuff but I believe Unicode
provides a lot of good data and guidance to work from. If you think my
proposal relies on one of those assumptions let me know.

To the extent that I do rely on any of those assumptions, it's mostly
to match libc's "C.UTF-8" behavior.

Regards,
	Jeff Davis