Message-ID: <ea061fe938602cd4321ed17742284ca86310defe.camel@j-davis.com>
Subject: Re: Small patch to improve safety of utf8_to_unicode().
From: Jeff Davis <pgsql@j-davis.com>
To: Tom Lane <tgl@sss.pgh.pa.us>, Chao Li <li.evan.chao@gmail.com>
Cc: pgsql-hackers@postgresql.org
Date: Mon, 13 Jul 2026 10:09:34 -0700
In-Reply-To: <3213927.1783950167@sss.pgh.pa.us>
References: <c355354e6c3f4a7aafb047361b73db247260fca0.camel@j-davis.com>
	 <541F240E-94AD-4D65-9794-7D6C316BC3FF@gmail.com>
	 <f11b6c89612f89b76b758807a32a1d3769652d90.camel@j-davis.com>
	 <C09D685C-FA5C-47E1-8EDB-B1FDD635309F@gmail.com>
	 <c4b27c82decf5b85523ad01f1df3785d8999f333.camel@j-davis.com>
	 <fbcb039a9ab3ba834f34174915254732fdcfae86.camel@j-davis.com>
	 <c82358d3b1bbd250a03937534bf76de9f4250d0c.camel@j-davis.com>
	 <fb7979177b7ab44d9abc2d423391de4ad9faea46.camel@j-davis.com>
	 <313BCA0E-2E74-4EB6-9AA8-10DB7F439151@gmail.com>
	 <592441c1d871a4ed0ac1708e132f447caa35869c.camel@j-davis.com>
	 <3058CAA1-40E0-4098-893B-615F77CD35E1@gmail.com>
	 <0cedb517aee8b79b8e31a4e42885ed88c2a67c5f.camel@j-davis.com>
	 <16CAD4DB-9D4A-481A-BB4E-78775C4F4B2C@gmail.com>
	 <f7000dc6842c75ace64656f9bf697a050fda0cfb.camel@j-davis.com>
	 <FAEA6374-2ED2-4988-8CDD-95157599024D@gmail.com>
	 <3213927.1783950167@sss.pgh.pa.us>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
User-Agent: Evolution 3.52.3-0ubuntu1.1 
MIME-Version: 1.0
Archived-At: 
 <https://www.postgresql.org/message-id/ea061fe938602cd4321ed17742284ca86310defe.camel%40j-davis.com>
Precedence: bulk

On Mon, 2026-07-13 at 09:42 -0400, Tom Lane wrote:
> What I think this is pointing out is that even if the input string's
> length fits in size_t, the length of the case-converted equivalent
> string might not.=C2=A0 The code won't write past dstsize, but it will
> try to compute the full output length required, and it won't notice
> if result_len overflows and wraps around.
>=20
> I would have written this off as an unreachable edge case, but our
> recent experience with unicode_normalize (cf. commit 066b7b144)
> makes me hesitant to assume that case-folding can't result in
> integer-multiple growth of the string length.=C2=A0 If it can, the
> overflow
> might be reachable on 32-bit platforms, resulting in a silently-
> broken
> result (but no memory clobber AFAICS).

From Unicode 17.0 section 5.18.2: "the maximum string expansion as a
result of case mapping in the Unicode Standard is three". Along with a
potential for 4X difference in byte length for a single code point in
UTF8, that makes a maximum of 12X expansion.

> Catering for this case seems like it would require messy warts on the
> casefolding functions' API.=C2=A0 So if we can convince ourselves that
> overflow can't really occur, I'd be content to add a comment
> demonstrating that.

Based on the actual data I expect the worst case to be smaller,
probably 3X. That would be convenient because I think that means it
can't overflow: the largest text value we support is 1GB, while 3GB
would still fit in a size_t. Unfortunately, even if that's true today,
it would be hard to be confident it won't change in a future version of
Unicode.

> =C2=A0 But if we can't prove that, I think we need some
> warts :-(

07211f64ac at least has an error path already, so we could set
*pconsumed=3D0 on overflow and return SIZE_MAX. It would be extra work
for the caller if it wants to distinguish between overflow and an
encoding error, but if we aren't expecting to hit it in practice, then
maybe it's OK to give an incorrect error.

Regards,
	Jeff Davis