Message-ID: <024a6d53f246c87ee2796563f930a66fde1c0c0d.camel@j-davis.com>
Subject: Re: C11: should we use char32_t for unicode code points?
From: Jeff Davis <pgsql@j-davis.com>
To: Thomas Munro <thomas.munro@gmail.com>
Cc: Tatsuo Ishii <ishii@postgresql.org>, pgsql-hackers@postgresql.org
Date: Wed, 29 Oct 2025 08:12:01 -0700
In-Reply-To: 
 <CA+hUKG+hDkp1etcfy=taxJ8ybf8KapyOjqdBRPF7yaoSoSj1_w@mail.gmail.com>
References: <bedcc93d06203dfd89815b10f815ca2de8626e85.camel@j-davis.com>
	 <20251024.184315.1449345234035166124.ishii@postgresql.org>
	 <435cf6bcf14bb91235da065c70d782863ca9b80e.camel@j-davis.com>
	 <CA+hUKGJ5Xh0KxLYXDZuPvw1_fHX=yuzb4xxtam1Cr6TPZZ1o+w@mail.gmail.com>
	 <efd8b17d7f97bc7653dde565e274fef32f5da65c.camel@j-davis.com>
	 <CA+hUKGLXQUYK7Cq5KbLGgTWo7pORs7yhBWO1AEnZt7xTYbLRhg@mail.gmail.com>
	 <8e5e4892c5c6b1b031b8e715dd254f21d9fb2bd9.camel@j-davis.com>
	 <CA+hUKG+hDkp1etcfy=taxJ8ybf8KapyOjqdBRPF7yaoSoSj1_w@mail.gmail.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
User-Agent: Evolution 3.52.3-0ubuntu1 
MIME-Version: 1.0
Archived-At: 
 <https://www.postgresql.org/message-id/024a6d53f246c87ee2796563f930a66fde1c0c0d.camel%40j-davis.com>
Precedence: bulk

On Wed, 2025-10-29 at 14:00 +1300, Thomas Munro wrote:
> I wonder if the logic to select the member/semantics could be turned
> into an enum in the encoding table, to make it even clearer, and then
> that could be used as an index into a table of ctype methods obejcts
> in _libc.c.

As long as we're able to isolate that logic in the libc provider,
that's reasonable. The other providers don't need that complexity, they
just need to decode straight to UTF-32.

> You showed char16_t for Windows, but we don't ever get char16_t out
> of
> wchar.c, it's always char32_t for UTF-8 input.=C2=A0 It's just that
> _libc.c
> truncates to UTF-16 or short-circuits to avoid overflow on that
> platform (and in the past AIX 32-bit and maybe more), so it wouldn't
> belong in a hypothetical union or enum.

Oh, I see.

> >=20
> Perhaps we could at least put the conversion in a new encoding table
> function pointer "pg_wchar_custom_to_wchar_t", so we could reserve a
> place to put that sort of optimisation in

That sounds like a good step forward. And maybe one to convert to UTF-
32 for ICU, also?

> If we do develop this idea though, one issue to contemplate is that
> EUC code points might generate more than one wchar_t, looking at
> EUC_JIS_2004[1].

Wow, that's unfortunate.


Regards,
	Jeff Davis