From: Jeff Davis <pgsql@j-davis.com>
To: Tom Lane <tgl@sss.pgh.pa.us>
Cc: Chao Li <li.evan.chao@gmail.com>
Cc: pgsql-hackers@postgresql.org
Subject: Re: Small patch to improve safety of utf8_to_unicode().
Date: Mon, 13 Jul 2026 20:26:47 -0700
Message-ID: <765830662fd26e0861189c775443fc76ea8213da.camel@j-davis.com> (raw)
In-Reply-To: <3228230.1783963239@sss.pgh.pa.us>
References: <c355354e6c3f4a7aafb047361b73db247260fca0.camel@j-davis.com>
<541F240E-94AD-4D65-9794-7D6C316BC3FF@gmail.com>
<f11b6c89612f89b76b758807a32a1d3769652d90.camel@j-davis.com>
<C09D685C-FA5C-47E1-8EDB-B1FDD635309F@gmail.com>
<c4b27c82decf5b85523ad01f1df3785d8999f333.camel@j-davis.com>
<fbcb039a9ab3ba834f34174915254732fdcfae86.camel@j-davis.com>
<c82358d3b1bbd250a03937534bf76de9f4250d0c.camel@j-davis.com>
<fb7979177b7ab44d9abc2d423391de4ad9faea46.camel@j-davis.com>
<313BCA0E-2E74-4EB6-9AA8-10DB7F439151@gmail.com>
<592441c1d871a4ed0ac1708e132f447caa35869c.camel@j-davis.com>
<3058CAA1-40E0-4098-893B-615F77CD35E1@gmail.com>
<0cedb517aee8b79b8e31a4e42885ed88c2a67c5f.camel@j-davis.com>
<16CAD4DB-9D4A-481A-BB4E-78775C4F4B2C@gmail.com>
<f7000dc6842c75ace64656f9bf697a050fda0cfb.camel@j-davis.com>
<FAEA6374-2ED2-4988-8CDD-95157599024D@gmail.com>
<3213927.1783950167@sss.pgh.pa.us>
<ea061fe938602cd4321ed17742284ca86310defe.camel@j-davis.com>
<3228110.1783963143@sss.pgh.pa.us>
<3228230.1783963239@sss.pgh.pa.us>
On Mon, 2026-07-13 at 13:20 -0400, Tom Lane wrote:
> I wrote:
> > Well, there aren't going to be any ASCII characters that produce 12
> > bytes, and probably not any LATIN1 characters either, so I think we
> > could assume that the shortest input character that could produce
> > such a long output would be 3 bytes.
>
> Ah sorry, I should have written BMP not LATIN1. Maybe it still holds
> across that set, but I'm less sure.
Attached a patch. I added some perl code to check that the expansion
factor of the UTF8 representation doesn't exceed 3, and added a static
assert in the C code to compare to MaxAllocSize.
Regards,
Jeff Davis
Attachments:v1-0001-Clarify-use-of-size_t-for-case-mapping.patch (text/x-patch, 5.4K) download
From 1fa065c524600609a495433069f43bdbb88e2e05 Mon Sep 17 00:00:00 2001
From: Jeff Davis <jeff@j-davis.com>
Date: Mon, 13 Jul 2026 19:31:56 -0700
Subject: [PATCH v1] Clarify use of size_t for case mapping.
It cannot overflow, but add some static checking to be sure.
Reported-by: Tom Lane <tgl@sss.pgh.pa.us>
Discussion: https://postgr.es/m/3213927.1783950167@sss.pgh.pa.us
---
src/backend/utils/adt/pg_locale_builtin.c | 10 ++++++
.../unicode/generate-unicode_case_table.pl | 32 +++++++++++++++++++
src/common/unicode_case.c | 6 ++++
src/include/common/unicode_case.h | 8 +++++
4 files changed, 56 insertions(+)
diff --git a/src/backend/utils/adt/pg_locale_builtin.c b/src/backend/utils/adt/pg_locale_builtin.cindex 5619daf43c3..c2cf68a518b 100644--- a/src/backend/utils/adt/pg_locale_builtin.c+++ b/src/backend/utils/adt/pg_locale_builtin.c@@ -17,9 +17,19 @@
#include "common/unicode_category.h"
#include "miscadmin.h"
#include "utils/builtins.h"
+#include "utils/memutils.h"
#include "utils/pg_locale.h"
#include "utils/syscache.h"
+/*+ * The largest text value must fit in MaxAllocSize, but then may grow during+ * case mapping. While the resulting string will not be representable as a new+ * text value, we must at least be sure not to overflow a size_t while+ * processing it.+ */+StaticAssertDecl(SIZE_MAX / MAX_CASE_UTF8_EXPANSION > MaxAllocSize,+ "case mapping may overflow size_t");+
extern pg_locale_t create_pg_locale_builtin(Oid collid,
MemoryContext context);
extern char *get_collation_actual_version_builtin(const char *collcollate);
diff --git a/src/common/unicode/generate-unicode_case_table.pl b/src/common/unicode/generate-unicode_case_table.plindex c1b94f47f56..b53b84ebc88 100644--- a/src/common/unicode/generate-unicode_case_table.pl+++ b/src/common/unicode/generate-unicode_case_table.pl@@ -11,6 +11,7 @@
use strict;
use warnings FATAL => 'all';
use Getopt::Long;
+use Encode qw(encode_utf8);
use FindBin;
use lib "$FindBin::RealBin/../../tools/";
@@ -25,6 +26,11 @@ my $output_table_file = "$output_path/unicode_case_table.h";
# of a single character. See Unicode section 5.18 "Case Mappings".
my $MAX_CASE_EXPANSION = 3;
++# The maximum expansion factor (in bytes) when case mapping a UTF8+# string.+my $MAX_CASE_UTF8_EXPANSION = 3;+
my $FH;
my %simple = ();
@@ -44,6 +50,10 @@ while (my $line = <$FH>)
die "Simple_Titlecase $code out of range" if $simple_titlecase > 0x10FFFF;
die "Simple_Uppercase $code out of range" if $simple_uppercase > 0x10FFFF;
+ check_utf8_expansion($code, ($simple_uppercase));+ check_utf8_expansion($code, ($simple_lowercase));+ check_utf8_expansion($code, ($simple_titlecase));+
if ($simple_lowercase || $simple_titlecase || $simple_uppercase)
{
$simple{$code} = {
@@ -113,6 +123,10 @@ while (my $line = <$FH>)
die "uppercase expansion for 0x$elts[0] exceeds maximum: '$elts[3]'"
if (scalar @upper) > $MAX_CASE_EXPANSION;
+ check_utf8_expansion($code, @lower);+ check_utf8_expansion($code, @title);+ check_utf8_expansion($code, @upper);+
# pad arrays to a fixed length of 3
while (scalar @upper < $MAX_CASE_EXPANSION) { push @upper, 0x000000 }
while (scalar @lower < $MAX_CASE_EXPANSION) { push @lower, 0x000000 }
@@ -174,6 +188,8 @@ while (my $line = <$FH>)
die "unsupported status type '$status'"
if $status ne 'S' && $status ne 'C' && $status ne 'F';
+ check_utf8_expansion($code, @fold);+
# initialize simple case mappings if they don't exist
$simple{$code} ||= {
Simple_Lowercase => $code,
@@ -522,6 +538,22 @@ EOS
close $OT;
+# compare byte length of UTF8 representation of the original codepoint+# to the codepoint(s) it maps to, and track the largest expansion+sub check_utf8_expansion+{+ my ($c, @a) = @_;++ my $utf8len1 = length(encode_utf8(chr $c));+ my $utf8len2 = length(encode_utf8(join '', map { chr } @a));+ my $factor = 0;+ $factor++ while ($factor * $utf8len1 < $utf8len2);++ my $c_hex = sprintf "0x%06x", $c;+ die "UTF8 expansion of $c_hex exceeds $MAX_CASE_UTF8_EXPANSION"+ if ($factor > $MAX_CASE_UTF8_EXPANSION);+}+
# The function generates C code with a series of nested if-else conditions
# to search for the matching interval.
sub branch
diff --git a/src/common/unicode_case.c b/src/common/unicode_case.cindex 399f08235a9..5f68154c8aa 100644--- a/src/common/unicode_case.c+++ b/src/common/unicode_case.c@@ -237,6 +237,12 @@ convert_case(char *dst, size_t dstsize, const char *src, size_t srclen,
size_t result_len = 0;
size_t boundary = 0;
+ /*+ * Must be guaranteed by caller to avoid overflow (text values limited to+ * MaxAllocSize anyway).+ */+ Assert(srclen < SIZE_MAX / MAX_CASE_UTF8_EXPANSION);+
Assert((str_casekind == CaseTitle && wbnext && wbstate) ||
(str_casekind != CaseTitle && !wbnext && !wbstate));
diff --git a/src/include/common/unicode_case.h b/src/include/common/unicode_case.hindex 1cbc0c14bc2..a2b752667ca 100644--- a/src/include/common/unicode_case.h+++ b/src/include/common/unicode_case.h@@ -14,6 +14,14 @@
#ifndef UNICODE_CASE_H
#define UNICODE_CASE_H
+/*+ * The maximum expansion factor (in bytes) when case mapping a UTF8 string.+ *+ * Not guaranteed by the standard, but checked while constructing the mapping+ * tables from Unicode data.+ */+#define MAX_CASE_UTF8_EXPANSION 3+
typedef size_t (*WordBoundaryNext) (void *wbstate);
char32_t unicode_lowercase_simple(char32_t code);
--
2.43.0
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: pgsql@j-davis.com, tgl@sss.pgh.pa.us, li.evan.chao@gmail.com
Subject: Re: Small patch to improve safety of utf8_to_unicode().
In-Reply-To: <765830662fd26e0861189c775443fc76ea8213da.camel@j-davis.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox