Message-ID: <765830662fd26e0861189c775443fc76ea8213da.camel@j-davis.com>
Subject: Re: Small patch to improve safety of utf8_to_unicode().
From: Jeff Davis <pgsql@j-davis.com>
To: Tom Lane <tgl@sss.pgh.pa.us>
Cc: Chao Li <li.evan.chao@gmail.com>, pgsql-hackers@postgresql.org
Date: Mon, 13 Jul 2026 20:26:47 -0700
In-Reply-To: <3228230.1783963239@sss.pgh.pa.us>
References: <c355354e6c3f4a7aafb047361b73db247260fca0.camel@j-davis.com>
	 <541F240E-94AD-4D65-9794-7D6C316BC3FF@gmail.com>
	 <f11b6c89612f89b76b758807a32a1d3769652d90.camel@j-davis.com>
	 <C09D685C-FA5C-47E1-8EDB-B1FDD635309F@gmail.com>
	 <c4b27c82decf5b85523ad01f1df3785d8999f333.camel@j-davis.com>
	 <fbcb039a9ab3ba834f34174915254732fdcfae86.camel@j-davis.com>
	 <c82358d3b1bbd250a03937534bf76de9f4250d0c.camel@j-davis.com>
	 <fb7979177b7ab44d9abc2d423391de4ad9faea46.camel@j-davis.com>
	 <313BCA0E-2E74-4EB6-9AA8-10DB7F439151@gmail.com>
	 <592441c1d871a4ed0ac1708e132f447caa35869c.camel@j-davis.com>
	 <3058CAA1-40E0-4098-893B-615F77CD35E1@gmail.com>
	 <0cedb517aee8b79b8e31a4e42885ed88c2a67c5f.camel@j-davis.com>
	 <16CAD4DB-9D4A-481A-BB4E-78775C4F4B2C@gmail.com>
	 <f7000dc6842c75ace64656f9bf697a050fda0cfb.camel@j-davis.com>
	 <FAEA6374-2ED2-4988-8CDD-95157599024D@gmail.com>
	 <3213927.1783950167@sss.pgh.pa.us>
	 <ea061fe938602cd4321ed17742284ca86310defe.camel@j-davis.com>
	 <3228110.1783963143@sss.pgh.pa.us> <3228230.1783963239@sss.pgh.pa.us>
Content-Type: multipart/mixed; boundary="=-qbF6kwIHlg25z2e514ot"
User-Agent: Evolution 3.52.3-0ubuntu1.1 
MIME-Version: 1.0
Archived-At: 
 <https://www.postgresql.org/message-id/765830662fd26e0861189c775443fc76ea8213da.camel%40j-davis.com>
Precedence: bulk

--=-qbF6kwIHlg25z2e514ot
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

On Mon, 2026-07-13 at 13:20 -0400, Tom Lane wrote:
> I wrote:
> > Well, there aren't going to be any ASCII characters that produce 12
> > bytes, and probably not any LATIN1 characters either, so I think we
> > could assume that the shortest input character that could produce
> > such a long output would be 3 bytes.
>=20
> Ah sorry, I should have written BMP not LATIN1.=C2=A0 Maybe it still hold=
s
> across that set, but I'm less sure.

Attached a patch. I added some perl code to check that the expansion
factor of the UTF8 representation doesn't exceed 3, and added a static
assert in the C code to compare to MaxAllocSize.

Regards,
	Jeff Davis


--=-qbF6kwIHlg25z2e514ot
Content-Disposition: attachment;
	filename="v1-0001-Clarify-use-of-size_t-for-case-mapping.patch"
Content-Type: text/x-patch;
 name="v1-0001-Clarify-use-of-size_t-for-case-mapping.patch";
	charset="UTF-8"
Content-Transfer-Encoding: base64

RnJvbSAxZmEwNjVjNTI0NjAwNjA5YTQ5NTQzMzA2OWY0M2JkYmI4OGUyZTA1IE1vbiBTZXAgMTcg
MDA6MDA6MDAgMjAwMQpGcm9tOiBKZWZmIERhdmlzIDxqZWZmQGotZGF2aXMuY29tPgpEYXRlOiBN
b24sIDEzIEp1bCAyMDI2IDE5OjMxOjU2IC0wNzAwClN1YmplY3Q6IFtQQVRDSCB2MV0gQ2xhcmlm
eSB1c2Ugb2Ygc2l6ZV90IGZvciBjYXNlIG1hcHBpbmcuCgpJdCBjYW5ub3Qgb3ZlcmZsb3csIGJ1
dCBhZGQgc29tZSBzdGF0aWMgY2hlY2tpbmcgdG8gYmUgc3VyZS4KClJlcG9ydGVkLWJ5OiBUb20g
TGFuZSA8dGdsQHNzcy5wZ2gucGEudXM+CkRpc2N1c3Npb246IGh0dHBzOi8vcG9zdGdyLmVzL20v
MzIxMzkyNy4xNzgzOTUwMTY3QHNzcy5wZ2gucGEudXMKLS0tCiBzcmMvYmFja2VuZC91dGlscy9h
ZHQvcGdfbG9jYWxlX2J1aWx0aW4uYyAgICAgfCAxMCArKysrKysKIC4uLi91bmljb2RlL2dlbmVy
YXRlLXVuaWNvZGVfY2FzZV90YWJsZS5wbCAgICB8IDMyICsrKysrKysrKysrKysrKysrKysKIHNy
Yy9jb21tb24vdW5pY29kZV9jYXNlLmMgICAgICAgICAgICAgICAgICAgICB8ICA2ICsrKysKIHNy
Yy9pbmNsdWRlL2NvbW1vbi91bmljb2RlX2Nhc2UuaCAgICAgICAgICAgICB8ICA4ICsrKysrCiA0
IGZpbGVzIGNoYW5nZWQsIDU2IGluc2VydGlvbnMoKykKCmRpZmYgLS1naXQgYS9zcmMvYmFja2Vu
ZC91dGlscy9hZHQvcGdfbG9jYWxlX2J1aWx0aW4uYyBiL3NyYy9iYWNrZW5kL3V0aWxzL2FkdC9w
Z19sb2NhbGVfYnVpbHRpbi5jCmluZGV4IDU2MTlkYWY0M2MzLi5jMmNmNjhhNTE4YiAxMDA2NDQK
LS0tIGEvc3JjL2JhY2tlbmQvdXRpbHMvYWR0L3BnX2xvY2FsZV9idWlsdGluLmMKKysrIGIvc3Jj
L2JhY2tlbmQvdXRpbHMvYWR0L3BnX2xvY2FsZV9idWlsdGluLmMKQEAgLTE3LDkgKzE3LDE5IEBA
CiAjaW5jbHVkZSAiY29tbW9uL3VuaWNvZGVfY2F0ZWdvcnkuaCIKICNpbmNsdWRlICJtaXNjYWRt
aW4uaCIKICNpbmNsdWRlICJ1dGlscy9idWlsdGlucy5oIgorI2luY2x1ZGUgInV0aWxzL21lbXV0
aWxzLmgiCiAjaW5jbHVkZSAidXRpbHMvcGdfbG9jYWxlLmgiCiAjaW5jbHVkZSAidXRpbHMvc3lz
Y2FjaGUuaCIKIAorLyoKKyAqIFRoZSBsYXJnZXN0IHRleHQgdmFsdWUgbXVzdCBmaXQgaW4gTWF4
QWxsb2NTaXplLCBidXQgdGhlbiBtYXkgZ3JvdyBkdXJpbmcKKyAqIGNhc2UgbWFwcGluZy4gV2hp
bGUgdGhlIHJlc3VsdGluZyBzdHJpbmcgd2lsbCBub3QgYmUgcmVwcmVzZW50YWJsZSBhcyBhIG5l
dworICogdGV4dCB2YWx1ZSwgd2UgbXVzdCBhdCBsZWFzdCBiZSBzdXJlIG5vdCB0byBvdmVyZmxv
dyBhIHNpemVfdCB3aGlsZQorICogcHJvY2Vzc2luZyBpdC4KKyAqLworU3RhdGljQXNzZXJ0RGVj
bChTSVpFX01BWCAvIE1BWF9DQVNFX1VURjhfRVhQQU5TSU9OID4gTWF4QWxsb2NTaXplLAorCQkJ
CSAiY2FzZSBtYXBwaW5nIG1heSBvdmVyZmxvdyBzaXplX3QiKTsKKwogZXh0ZXJuIHBnX2xvY2Fs
ZV90IGNyZWF0ZV9wZ19sb2NhbGVfYnVpbHRpbihPaWQgY29sbGlkLAogCQkJCQkJCQkJCQlNZW1v
cnlDb250ZXh0IGNvbnRleHQpOwogZXh0ZXJuIGNoYXIgKmdldF9jb2xsYXRpb25fYWN0dWFsX3Zl
cnNpb25fYnVpbHRpbihjb25zdCBjaGFyICpjb2xsY29sbGF0ZSk7CmRpZmYgLS1naXQgYS9zcmMv
Y29tbW9uL3VuaWNvZGUvZ2VuZXJhdGUtdW5pY29kZV9jYXNlX3RhYmxlLnBsIGIvc3JjL2NvbW1v
bi91bmljb2RlL2dlbmVyYXRlLXVuaWNvZGVfY2FzZV90YWJsZS5wbAppbmRleCBjMWI5NGY0N2Y1
Ni4uYjUzYjg0ZWJjODggMTAwNjQ0Ci0tLSBhL3NyYy9jb21tb24vdW5pY29kZS9nZW5lcmF0ZS11
bmljb2RlX2Nhc2VfdGFibGUucGwKKysrIGIvc3JjL2NvbW1vbi91bmljb2RlL2dlbmVyYXRlLXVu
aWNvZGVfY2FzZV90YWJsZS5wbApAQCAtMTEsNiArMTEsNyBAQAogdXNlIHN0cmljdDsKIHVzZSB3
YXJuaW5ncyBGQVRBTCA9PiAnYWxsJzsKIHVzZSBHZXRvcHQ6Okxvbmc7Cit1c2UgRW5jb2RlIHF3
KGVuY29kZV91dGY4KTsKIAogdXNlIEZpbmRCaW47CiB1c2UgbGliICIkRmluZEJpbjo6UmVhbEJp
bi8uLi8uLi90b29scy8iOwpAQCAtMjUsNiArMjYsMTEgQEAgbXkgJG91dHB1dF90YWJsZV9maWxl
ID0gIiRvdXRwdXRfcGF0aC91bmljb2RlX2Nhc2VfdGFibGUuaCI7CiAjIG9mIGEgc2luZ2xlIGNo
YXJhY3Rlci4gU2VlIFVuaWNvZGUgc2VjdGlvbiA1LjE4ICJDYXNlIE1hcHBpbmdzIi4KIG15ICRN
QVhfQ0FTRV9FWFBBTlNJT04gPSAzOwogCisKKyMgVGhlIG1heGltdW0gZXhwYW5zaW9uIGZhY3Rv
ciAoaW4gYnl0ZXMpIHdoZW4gY2FzZSBtYXBwaW5nIGEgVVRGOAorIyBzdHJpbmcuCitteSAkTUFY
X0NBU0VfVVRGOF9FWFBBTlNJT04gPSAzOworCiBteSAkRkg7CiAKIG15ICVzaW1wbGUgPSAoKTsK
QEAgLTQ0LDYgKzUwLDEwIEBAIHdoaWxlIChteSAkbGluZSA9IDwkRkg+KQogCWRpZSAiU2ltcGxl
X1RpdGxlY2FzZSAkY29kZSBvdXQgb2YgcmFuZ2UiIGlmICRzaW1wbGVfdGl0bGVjYXNlID4gMHgx
MEZGRkY7CiAJZGllICJTaW1wbGVfVXBwZXJjYXNlICRjb2RlIG91dCBvZiByYW5nZSIgaWYgJHNp
bXBsZV91cHBlcmNhc2UgPiAweDEwRkZGRjsKIAorCWNoZWNrX3V0ZjhfZXhwYW5zaW9uKCRjb2Rl
LCAoJHNpbXBsZV91cHBlcmNhc2UpKTsKKwljaGVja191dGY4X2V4cGFuc2lvbigkY29kZSwgKCRz
aW1wbGVfbG93ZXJjYXNlKSk7CisJY2hlY2tfdXRmOF9leHBhbnNpb24oJGNvZGUsICgkc2ltcGxl
X3RpdGxlY2FzZSkpOworCiAJaWYgKCRzaW1wbGVfbG93ZXJjYXNlIHx8ICRzaW1wbGVfdGl0bGVj
YXNlIHx8ICRzaW1wbGVfdXBwZXJjYXNlKQogCXsKIAkJJHNpbXBsZXskY29kZX0gPSB7CkBAIC0x
MTMsNiArMTIzLDEwIEBAIHdoaWxlIChteSAkbGluZSA9IDwkRkg+KQogCWRpZSAidXBwZXJjYXNl
IGV4cGFuc2lvbiBmb3IgMHgkZWx0c1swXSBleGNlZWRzIG1heGltdW06ICckZWx0c1szXSciCiAJ
ICBpZiAoc2NhbGFyIEB1cHBlcikgPiAkTUFYX0NBU0VfRVhQQU5TSU9OOwogCisJY2hlY2tfdXRm
OF9leHBhbnNpb24oJGNvZGUsIEBsb3dlcik7CisJY2hlY2tfdXRmOF9leHBhbnNpb24oJGNvZGUs
IEB0aXRsZSk7CisJY2hlY2tfdXRmOF9leHBhbnNpb24oJGNvZGUsIEB1cHBlcik7CisKIAkjIHBh
ZCBhcnJheXMgdG8gYSBmaXhlZCBsZW5ndGggb2YgMwogCXdoaWxlIChzY2FsYXIgQHVwcGVyIDwg
JE1BWF9DQVNFX0VYUEFOU0lPTikgeyBwdXNoIEB1cHBlciwgMHgwMDAwMDAgfQogCXdoaWxlIChz
Y2FsYXIgQGxvd2VyIDwgJE1BWF9DQVNFX0VYUEFOU0lPTikgeyBwdXNoIEBsb3dlciwgMHgwMDAw
MDAgfQpAQCAtMTc0LDYgKzE4OCw4IEBAIHdoaWxlIChteSAkbGluZSA9IDwkRkg+KQogCWRpZSAi
dW5zdXBwb3J0ZWQgc3RhdHVzIHR5cGUgJyRzdGF0dXMnIgogCSAgaWYgJHN0YXR1cyBuZSAnUycg
JiYgJHN0YXR1cyBuZSAnQycgJiYgJHN0YXR1cyBuZSAnRic7CiAKKwljaGVja191dGY4X2V4cGFu
c2lvbigkY29kZSwgQGZvbGQpOworCiAJIyBpbml0aWFsaXplIHNpbXBsZSBjYXNlIG1hcHBpbmdz
IGlmIHRoZXkgZG9uJ3QgZXhpc3QKIAkkc2ltcGxleyRjb2RlfSB8fD0gewogCQlTaW1wbGVfTG93
ZXJjYXNlID0+ICRjb2RlLApAQCAtNTIyLDYgKzUzOCwyMiBAQCBFT1MKIAogY2xvc2UgJE9UOwog
CisjIGNvbXBhcmUgYnl0ZSBsZW5ndGggb2YgVVRGOCByZXByZXNlbnRhdGlvbiBvZiB0aGUgb3Jp
Z2luYWwgY29kZXBvaW50CisjIHRvIHRoZSBjb2RlcG9pbnQocykgaXQgbWFwcyB0bywgYW5kIHRy
YWNrIHRoZSBsYXJnZXN0IGV4cGFuc2lvbgorc3ViIGNoZWNrX3V0ZjhfZXhwYW5zaW9uCit7CisJ
bXkgKCRjLCBAYSkgPSBAXzsKKworCW15ICR1dGY4bGVuMSA9IGxlbmd0aChlbmNvZGVfdXRmOChj
aHIgJGMpKTsKKwlteSAkdXRmOGxlbjIgPSBsZW5ndGgoZW5jb2RlX3V0Zjgoam9pbiAnJywgbWFw
IHsgY2hyIH0gQGEpKTsKKwlteSAkZmFjdG9yID0gMDsKKwkkZmFjdG9yKysgd2hpbGUgKCRmYWN0
b3IgKiAkdXRmOGxlbjEgPCAkdXRmOGxlbjIpOworCisJbXkgJGNfaGV4ID0gc3ByaW50ZiAiMHgl
MDZ4IiwgJGM7CisJZGllICJVVEY4IGV4cGFuc2lvbiBvZiAkY19oZXggZXhjZWVkcyAkTUFYX0NB
U0VfVVRGOF9FWFBBTlNJT04iCisJICBpZiAoJGZhY3RvciA+ICRNQVhfQ0FTRV9VVEY4X0VYUEFO
U0lPTik7Cit9CisKICMgVGhlIGZ1bmN0aW9uIGdlbmVyYXRlcyBDIGNvZGUgd2l0aCBhIHNlcmll
cyBvZiBuZXN0ZWQgaWYtZWxzZSBjb25kaXRpb25zCiAjIHRvIHNlYXJjaCBmb3IgdGhlIG1hdGNo
aW5nIGludGVydmFsLgogc3ViIGJyYW5jaApkaWZmIC0tZ2l0IGEvc3JjL2NvbW1vbi91bmljb2Rl
X2Nhc2UuYyBiL3NyYy9jb21tb24vdW5pY29kZV9jYXNlLmMKaW5kZXggMzk5ZjA4MjM1YTkuLjVm
NjgxNTRjOGFhIDEwMDY0NAotLS0gYS9zcmMvY29tbW9uL3VuaWNvZGVfY2FzZS5jCisrKyBiL3Ny
Yy9jb21tb24vdW5pY29kZV9jYXNlLmMKQEAgLTIzNyw2ICsyMzcsMTIgQEAgY29udmVydF9jYXNl
KGNoYXIgKmRzdCwgc2l6ZV90IGRzdHNpemUsIGNvbnN0IGNoYXIgKnNyYywgc2l6ZV90IHNyY2xl
biwKIAlzaXplX3QJCXJlc3VsdF9sZW4gPSAwOwogCXNpemVfdAkJYm91bmRhcnkgPSAwOwogCisJ
LyoKKwkgKiBNdXN0IGJlIGd1YXJhbnRlZWQgYnkgY2FsbGVyIHRvIGF2b2lkIG92ZXJmbG93ICh0
ZXh0IHZhbHVlcyBsaW1pdGVkIHRvCisJICogTWF4QWxsb2NTaXplIGFueXdheSkuCisJICovCisJ
QXNzZXJ0KHNyY2xlbiA8IFNJWkVfTUFYIC8gTUFYX0NBU0VfVVRGOF9FWFBBTlNJT04pOworCiAJ
QXNzZXJ0KChzdHJfY2FzZWtpbmQgPT0gQ2FzZVRpdGxlICYmIHdibmV4dCAmJiB3YnN0YXRlKSB8
fAogCQkgICAoc3RyX2Nhc2VraW5kICE9IENhc2VUaXRsZSAmJiAhd2JuZXh0ICYmICF3YnN0YXRl
KSk7CiAKZGlmZiAtLWdpdCBhL3NyYy9pbmNsdWRlL2NvbW1vbi91bmljb2RlX2Nhc2UuaCBiL3Ny
Yy9pbmNsdWRlL2NvbW1vbi91bmljb2RlX2Nhc2UuaAppbmRleCAxY2JjMGMxNGJjMi4uYTJiNzUy
NjY3Y2EgMTAwNjQ0Ci0tLSBhL3NyYy9pbmNsdWRlL2NvbW1vbi91bmljb2RlX2Nhc2UuaAorKysg
Yi9zcmMvaW5jbHVkZS9jb21tb24vdW5pY29kZV9jYXNlLmgKQEAgLTE0LDYgKzE0LDE0IEBACiAj
aWZuZGVmIFVOSUNPREVfQ0FTRV9ICiAjZGVmaW5lIFVOSUNPREVfQ0FTRV9ICiAKKy8qCisgKiBU
aGUgbWF4aW11bSBleHBhbnNpb24gZmFjdG9yIChpbiBieXRlcykgd2hlbiBjYXNlIG1hcHBpbmcg
YSBVVEY4IHN0cmluZy4KKyAqCisgKiBOb3QgZ3VhcmFudGVlZCBieSB0aGUgc3RhbmRhcmQsIGJ1
dCBjaGVja2VkIHdoaWxlIGNvbnN0cnVjdGluZyB0aGUgbWFwcGluZworICogdGFibGVzIGZyb20g
VW5pY29kZSBkYXRhLgorICovCisjZGVmaW5lIE1BWF9DQVNFX1VURjhfRVhQQU5TSU9OIDMKKwog
dHlwZWRlZiBzaXplX3QgKCpXb3JkQm91bmRhcnlOZXh0KSAodm9pZCAqd2JzdGF0ZSk7CiAKIGNo
YXIzMl90CXVuaWNvZGVfbG93ZXJjYXNlX3NpbXBsZShjaGFyMzJfdCBjb2RlKTsKLS0gCjIuNDMu
MAoK


--=-qbF6kwIHlg25z2e514ot--