Message-ID: <c4b27c82decf5b85523ad01f1df3785d8999f333.camel@j-davis.com>
Subject: Re: Small patch to improve safety of utf8_to_unicode().
From: Jeff Davis <pgsql@j-davis.com>
To: Chao Li <li.evan.chao@gmail.com>
Cc: pgsql-hackers@postgresql.org
Date: Wed, 17 Dec 2025 11:37:59 -0800
In-Reply-To: <C09D685C-FA5C-47E1-8EDB-B1FDD635309F@gmail.com>
References: <c355354e6c3f4a7aafb047361b73db247260fca0.camel@j-davis.com>
	 <541F240E-94AD-4D65-9794-7D6C316BC3FF@gmail.com>
	 <f11b6c89612f89b76b758807a32a1d3769652d90.camel@j-davis.com>
	 <C09D685C-FA5C-47E1-8EDB-B1FDD635309F@gmail.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
User-Agent: Evolution 3.52.3-0ubuntu1.1 
MIME-Version: 1.0
Archived-At: 
 <https://www.postgresql.org/message-id/c4b27c82decf5b85523ad01f1df3785d8999f333.camel%40j-davis.com>
Precedence: bulk

On Tue, 2025-12-16 at 07:34 +0800, Chao Li wrote:
> > <v2-0001-Make-utf8_to_unicode-safer.patch>
>=20
> V2 LGTM.

On second thought, if we're going to change something here, we should
probably have a more flexible API for both utf8_to_unicode() and
unicode_to_utf8().

Looking at the callers, I think we want to have signatures something
like:

/* returns number of bytes consumed, or -1 */
static inline ssize_t
utf8_to_unicode(char32_t *cp, const unsigned char *src, size_t srclen)
{
    ...
}

/* returns number of bytes written, or -1 */
static inline ssize_t
unicode_to_utf8(unsigned char *dst, size_t dstsize, char32_t cp)
{
    ...
}

That would make both APIs safer, and the caller wouldn't need to call
unicode_utf8len() or pg_utf8_mblen() separately.

We could also do more validation, but of course then the callers would
need to do something if they encounter a failure. We could also try to
catch NUL terminators in the middle of a sequence, which might be
useful.

Regards,
	Jeff Davis