Message-ID: <0cedb517aee8b79b8e31a4e42885ed88c2a67c5f.camel@j-davis.com>
Subject: Re: Small patch to improve safety of utf8_to_unicode().
From: Jeff Davis <pgsql@j-davis.com>
To: Chao Li <li.evan.chao@gmail.com>
Cc: pgsql-hackers@postgresql.org
Date: Thu, 25 Jun 2026 10:38:40 -0700
In-Reply-To: <3058CAA1-40E0-4098-893B-615F77CD35E1@gmail.com>
References: <c355354e6c3f4a7aafb047361b73db247260fca0.camel@j-davis.com>
	 <541F240E-94AD-4D65-9794-7D6C316BC3FF@gmail.com>
	 <f11b6c89612f89b76b758807a32a1d3769652d90.camel@j-davis.com>
	 <C09D685C-FA5C-47E1-8EDB-B1FDD635309F@gmail.com>
	 <c4b27c82decf5b85523ad01f1df3785d8999f333.camel@j-davis.com>
	 <fbcb039a9ab3ba834f34174915254732fdcfae86.camel@j-davis.com>
	 <c82358d3b1bbd250a03937534bf76de9f4250d0c.camel@j-davis.com>
	 <fb7979177b7ab44d9abc2d423391de4ad9faea46.camel@j-davis.com>
	 <313BCA0E-2E74-4EB6-9AA8-10DB7F439151@gmail.com>
	 <592441c1d871a4ed0ac1708e132f447caa35869c.camel@j-davis.com>
	 <3058CAA1-40E0-4098-893B-615F77CD35E1@gmail.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
User-Agent: Evolution 3.52.3-0ubuntu1.1 
MIME-Version: 1.0
Archived-At: 
 <https://www.postgresql.org/message-id/0cedb517aee8b79b8e31a4e42885ed88c2a67c5f.camel%40j-davis.com>
Precedence: bulk

On Thu, 2026-06-25 at 12:10 +0800, Chao Li wrote:
> This uses the second byte, \x20, without validating. So it looks like
> the patch prevents reading past the end of the string, but it may not
> fully defend against invalid UTF-8 sequences.

Correct. We don't do full UTF8 validation until the last patch in the
series, which is not being backported.

Trying to do full validation in the backbranches seems more likely to
cause problems than prevent them. We aren't expecting invalid UTF8, but
in the event it got there somehow (perhaps from an old upgraded
instance), throwing errors after a minor release is probably not
helpful.

Even in master, I am not 100% sure we want to detect other kinds of
validation errors while processing the UTF8. By the time we are using
the value, maybe truncated multibyte sequences are the only thing we
care about, and we just need to be sure the code can handle anything
that fits in a char32_t.

Another thing to consider is an embedded NUL character, which is valid
UTF8 but not valid in a TEXT value.

Regards,
	Jeff Davis