mail-alias-created-date: 1748907026120
Content-Type: text/plain;
	charset=utf-8
Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3864.600.51.1.1\))
Subject: Re: [PATCH] Fix recognizing 0x11A7 as a Hangul T syllable in Unicode
 normalization
From: Diego Frias <mail@dzfrias.dev>
In-Reply-To: <aiD55PbCxUKuScRr@paquier.xyz>
Date: Thu, 4 Jun 2026 09:32:53 -0700
Cc: pgsql-hackers@lists.postgresql.org
Content-Transfer-Encoding: quoted-printable
Message-Id: <D6D525DE-C1F2-498D-829C-396240337B59@dzfrias.dev>
References: <B92ED640-7D4A-4505-B09F-3548F58CBB16@dzfrias.dev>
 <aiD55PbCxUKuScRr@paquier.xyz>
To: Michael Paquier <michael@paquier.xyz>
Archived-At: 
 <https://www.postgresql.org/message-id/D6D525DE-C1F2-498D-829C-396240337B59%40dzfrias.dev>
Precedence: bulk

Looks great! Thanks for letting me know where the tests live. I=E2=80=99ll=

try to get these tests in the official Unicode test suite, too. Should
help future implementors.

Thanks,
Diego

> On Jun 3, 2026, at 9:07=E2=80=AFPM, Michael Paquier =
<michael@paquier.xyz> wrote:
>=20
> On Mon, Jun 01, 2026 at 11:38:32AM -0700, Diego Frias wrote:
>> In short, TCount actually counts 1 more than the number of T
>> syllables; this is so s % TCount =3D=3D 0 implies that s has no T
>> syllable (because the 0th place represents the absence of a T
>> syllable), where s is the s-index of a precomposed Hangul
>> character. Anyway, since PostgreSQL recognizes 0x11A7 as a T
>> syllable, the composition algorithm yields a nonsense character when
>> 0x11A7 is put in the T position.
>=20
> Oops.  Yes, including TBASE in the recomposition is incorrect, finding
> your quote here (TBase is set to one less..):
> https://unicode.org/versions/Unicode17.0.0/core-spec/chapter-3/#G59688
>=20
> The character gets eaten by the normalization.  Pas glop.
>=20
>> Let me know if this patch needs anything else. I can write a test
>> for this, but it looks like the current testing setup in
>> src/common/norm_test.c only runs the Unicode test suite and isn=E2=80=99=
t
>> built for writing custom tests. If that is something of interest,
>> though, I=E2=80=99m happy to add that to this patch.
>=20
> We have a set of tests in src/test/regress/sql/unicode.sql that would
> fit nicely with what you want to address here.  For this specific
> problem, this would work:
> SELECT normalize(U&'\AC00\11A7', NFC) =3D U&'\AC00\11A7';
>=20
> How about adding more normalization check patterns, while on it?  I am
> finishing with the attached, all things combined.  Diego. what do you
> think?
> --
> Michael
> <0001-Fix-off-by-one-with-NFC-recomposition-for-Hangul-U-1.patch>