Message-ID: <e620dc508987e27dc542657d39da02f65bcaedcb.camel@j-davis.com>
Subject: Re: Small patch to improve safety of utf8_to_unicode().
From: Jeff Davis <pgsql@j-davis.com>
To: Ayush Tiwari <ayushtiwari.slg01@gmail.com>
Cc: Chao Li <li.evan.chao@gmail.com>, pgsql-hackers@postgresql.org
Date: Wed, 24 Jun 2026 16:43:28 -0700
In-Reply-To: 
 <CAJTYsWUoYh84OugcnkghK24pLFsqxTy2Ajq7druCYVne_Dj8gw@mail.gmail.com>
References: <c355354e6c3f4a7aafb047361b73db247260fca0.camel@j-davis.com>
	 <541F240E-94AD-4D65-9794-7D6C316BC3FF@gmail.com>
	 <f11b6c89612f89b76b758807a32a1d3769652d90.camel@j-davis.com>
	 <C09D685C-FA5C-47E1-8EDB-B1FDD635309F@gmail.com>
	 <c4b27c82decf5b85523ad01f1df3785d8999f333.camel@j-davis.com>
	 <fbcb039a9ab3ba834f34174915254732fdcfae86.camel@j-davis.com>
	 <c82358d3b1bbd250a03937534bf76de9f4250d0c.camel@j-davis.com>
	 <fb7979177b7ab44d9abc2d423391de4ad9faea46.camel@j-davis.com>
	 <CAJTYsWUoYh84OugcnkghK24pLFsqxTy2Ajq7druCYVne_Dj8gw@mail.gmail.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
User-Agent: Evolution 3.52.3-0ubuntu1.1 
MIME-Version: 1.0
Archived-At: 
 <https://www.postgresql.org/message-id/e620dc508987e27dc542657d39da02f65bcaedcb.camel%40j-davis.com>
Precedence: bulk

On Wed, 2026-06-24 at 14:59 +0530, Ayush Tiwari wrote:
> I took a look at the v5 series and tried it locally. The split
> between the
> backpatchable defensive change, the final-sigma fix, and the newer
> utf8encode/utf8decode API work for master makes sense to me.=C2=A0

Thank you for taking a look.

> On my machine, case-check fails before reaching the new invalid-UTF8
> assertions. I think it's because PostgreSQL's Unicode tables are 17.0
> while my
> system ICU is 15.1, so the exhaustive ICU comparison in test_icu()
> hits a
> changed mapping and exits first.

Yes, that test is a bit awkward because most of it is a comparison to
results from ICU, so if ICU is unavailable or based on an older version
of Unicode, then the test doesn't work.

> Would it be worth running test_convert_case() before
> test_icu()?

If we want to make those independent of ICU, I think we'd move them to
a regular test suite that's exercised everywhere (not just as part of
the 'update-unicode' target). But if we did so, that would be a very
small test suite, because most of the results are already checked in
the normal SQL tests. What makes these tests different is that they are
exercising invalid UTF8 behavior, which we don't expect to happen
through ordinary SQL.

Regards,
	Jeff Davis