MIME-Version: 1.0
References: <c355354e6c3f4a7aafb047361b73db247260fca0.camel@j-davis.com>
 <541F240E-94AD-4D65-9794-7D6C316BC3FF@gmail.com>
 <f11b6c89612f89b76b758807a32a1d3769652d90.camel@j-davis.com>
 <C09D685C-FA5C-47E1-8EDB-B1FDD635309F@gmail.com>
 <c4b27c82decf5b85523ad01f1df3785d8999f333.camel@j-davis.com>
 <fbcb039a9ab3ba834f34174915254732fdcfae86.camel@j-davis.com>
 <c82358d3b1bbd250a03937534bf76de9f4250d0c.camel@j-davis.com>
 <fb7979177b7ab44d9abc2d423391de4ad9faea46.camel@j-davis.com>
In-Reply-To: <fb7979177b7ab44d9abc2d423391de4ad9faea46.camel@j-davis.com>
From: Ayush Tiwari <ayushtiwari.slg01@gmail.com>
Date: Wed, 24 Jun 2026 14:59:02 +0530
Message-ID: 
 <CAJTYsWUoYh84OugcnkghK24pLFsqxTy2Ajq7druCYVne_Dj8gw@mail.gmail.com>
Subject: Re: Small patch to improve safety of utf8_to_unicode().
To: Jeff Davis <pgsql@j-davis.com>
Cc: Chao Li <li.evan.chao@gmail.com>, pgsql-hackers@postgresql.org
Content-Type: multipart/alternative; boundary="00000000000038e9920654fc8240"
Archived-At: 
 <https://www.postgresql.org/message-id/CAJTYsWUoYh84OugcnkghK24pLFsqxTy2Ajq7druCYVne_Dj8gw%40mail.gmail.com>
Precedence: bulk

--00000000000038e9920654fc8240
Content-Type: text/plain; charset="UTF-8"

Hi,

On Wed, 24 Jun 2026 at 11:15, Jeff Davis <pgsql@j-davis.com> wrote:

> On Mon, 2026-06-22 at 19:02 -0700, Jeff Davis wrote:
> > v4 attached.
>
> v5 attached.
>
> There's an extra patch 0002 to fix a logic bug when handling final
> sigma (only affects the builtin pg_unicode_fast locale), which I think
> should be backported to 18.
>
> Also added tests.
>

Thanks for the patch!

I took a look at the v5 series and tried it locally. The split between the
backpatchable defensive change, the final-sigma fix, and the newer
utf8encode/utf8decode API work for master makes sense to me.

On my machine, case-check fails before reaching the new invalid-UTF8
assertions. I think it's because PostgreSQL's Unicode tables are 17.0 while
my
system ICU is 15.1, so the exhaustive ICU comparison in test_icu() hits a
changed mapping and exits first.

Would it be worth running test_convert_case() before
test_icu()? It wouldn't make case-check pass on a mismatched-ICU system,
but it
would at least let the non-ICU conversion and the new invalid-UTF8 cases run
before the ICU comparison aborts.

Regards,
Ayush

--00000000000038e9920654fc8240
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div dir=3D"ltr"><div dir=3D"ltr">Hi,</div><br><div class=
=3D"gmail_quote"><div dir=3D"ltr" class=3D"gmail_attr">On Wed, 24 Jun 2026 =
at 11:15, Jeff Davis &lt;<a href=3D"mailto:pgsql@j-davis.com" target=3D"_bl=
ank">pgsql@j-davis.com</a>&gt; wrote:<br></div><blockquote class=3D"gmail_q=
uote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,2=
04);padding-left:1ex">On Mon, 2026-06-22 at 19:02 -0700, Jeff Davis wrote:<=
br>
&gt; v4 attached.<br>
<br>
v5 attached.<br>
<br>
There&#39;s an extra patch 0002 to fix a logic bug when handling final<br>
sigma (only affects the builtin pg_unicode_fast locale), which I think<br>
should be backported to 18.<br>
<br>
Also added tests.<br></blockquote><div><br>Thanks for the patch!<br><br>I t=
ook a look at the v5 series and tried it locally. The split between the<br>=
backpatchable defensive change, the final-sigma fix, and the newer<br>utf8e=
ncode/utf8decode API work for master makes sense to me.=C2=A0<br><br>On my =
machine, case-check fails before reaching the new invalid-UTF8<br>assertion=
s. I think it&#39;s because PostgreSQL&#39;s Unicode tables are 17.0 while =
my<br>system ICU is 15.1, so the exhaustive ICU comparison in test_icu() hi=
ts a</div><div>changed mapping and exits first.<br><br>Would it be worth ru=
nning test_convert_case() before<br>test_icu()? It wouldn&#39;t make case-c=
heck pass on a mismatched-ICU system, but it<br>would at least let the non-=
ICU conversion and the new invalid-UTF8 cases run<br>before the ICU compari=
son aborts.<br><br>Regards,<br>Ayush</div></div></div>
</div>

--00000000000038e9920654fc8240--