Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.98.2) (envelope-from ) id 1x7bwz-00000000W2P-1HB6 for pgsql-hackers@arkaria.postgresql.org; Fri, 18 Sep 2026 17:00:37 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.98.2) (envelope-from ) id 1x7bwy-00000002TMo-2cHa for pgsql-hackers@arkaria.postgresql.org; Fri, 18 Sep 2026 17:00:36 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.98.2) (envelope-from ) id 1x7bwx-00000002TMf-49Tn for pgsql-hackers@lists.postgresql.org; Fri, 18 Sep 2026 17:00:36 +0000 Received: from fhigh-b3-smtp.messagingengine.com ([202.12.124.154]) by makus.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.98.2) (envelope-from ) id 1x7bwv-000000003st-1uDn for pgsql-hackers@lists.postgresql.org; Fri, 18 Sep 2026 17:00:35 +0000 Received: from phl-compute-05.internal (phl-compute-05.internal [10.202.2.45]) by mailfhigh.stl.internal (Postfix) with ESMTP id 724DB7A00E6 for ; Fri, 18 Sep 2026 13:00:33 -0400 (EDT) Received: from phl-imap-15 ([10.202.2.104]) by phl-compute-05.internal (MEProxy); Fri, 18 Sep 2026 13:00:33 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=partin.io; h=cc :content-type:content-type:date:date:from:from:in-reply-to :message-id:mime-version:reply-to:subject:subject:to:to; s=fm1; t=1789750833; x=1789837233; bh=i0Md4djynmkdl90tYs0hBpQtdg0hpN5A gkprodofTmQ=; b=aHXeSur1Vsj8BFYupNDJ2+NvFgYZdWvgO2lZFdNxuxzEVOlq LW2CeZ9WQ2IDf8qyhJgCO+ZtVk/ZCRGDd0Ocz1I1ax0yPlxKCi5fbXzDwR8StEGF NAJjNP9AyN+5k9Chm+tbImg0ODeSVLLXt6Pns3gIqn+EvcXGGBlFfVEXHKhV4WOo xZ/gxOZpoXz1MB+nw+HPbvdK8cRPJHQE03k1G3sjxaVEFSGWAhNNtRIugWjb4but qFGLrr2xfgV/9ltfqoe5eYOLFSDDb4F4/EZk/d6uKvK3+CZbLI5nP3rgytpP21lB QvZ5+iLnVtzOnbQS4sM5HScyz3vrfNvFmOpysQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:message-id :mime-version:reply-to:subject:subject:to:to:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm1; t=1789750833; x= 1789837233; bh=i0Md4djynmkdl90tYs0hBpQtdg0hpN5AgkprodofTmQ=; b=o OcKoyzrW7S8JfZeXmjSHeppvWFGS3BIg+/S+0RRIOiBtEsDCvWwa2xK4hJF9+R7h LX1az9fkjyBjbZ46en1NA5uDC6AItU6IGX2vnTl8QgV26Rc0xykOkYC0u0PpMoii FQWvCeScSI0t/YAtb9J2CWtrAhyJPJE5bFcagD+zbtQ4V14xT0H5IFcdpKkMhU6a J5d727E/bAJSh9IFd6DT9DlmsQ2gx+5MERuaJCBckmSjkedksUJUjiCbnHxB8Wmu DlO/I4+RZrSUtxBeJJBn+wRWZIoPDmk76VSWF+Jeqp2eDiag6ClNxbzxNJ168zQI KqxJEWK/9vOtdMCTb2OUg== X-ME-Sender: X-ME-Proxy-Cause: dmFkZTFKlHUWZa4E+57lcSIBQQTTQ8OyGdfP8YgbFs7EcgrjpI4u5VHF88Fq/qzTDKzvyD GxtLVEZWo/DIVI1L0gv0Wc3TXBpuQQOWg7Y9R1I8W8zjvn+GN0sT1gOT2SXvhvoU2MocrD JCUoA3TQMICVxTDtWSrPJmItrylWKnxLlL/hxRR3UmbE7j6DW8tGm52KaETok/XvWnjMfj hEqAvIsBaStK6jQqGNC7NoSF/SMS37B14GZHvX7WPDzgWcMYCZQ2SdaQiY7QL2djBai50I WAQ0O7FsNow7yX/wk+rpxb4YL5q+7HJotyRdXfnUxm5aRx8FXKxD75zfk6G2abwoeAAuA6 2jzby/6IKisMSyAqxtxbugFyL4WdnRqrNJoJFEocAgfyQmRnfwfz9YuNcuTs27FKNWtzIo LpnN/hw2/qfSp1pKbmTWd5UhA6pBagBC6Cn3DrGzeY4Q/nka+O9h9qKvRU1T1HQ7kiVIwm jKTFwCatKmi31MdrXpiRdChUotrKgUrt9jF7zYUOao7TqmZO/iXWmGrQi4wz7dgZpotipA 7bBbxzz2jNuFzAHzioEhdVgfr05FxTBTbrYiZq3dhno6OPSQCZgCfqWWJt+bAG682VjAcV maaxkwpe/JJ/6dGbS56btGnUaX9XD8vzt3AcE7pgirvwIf+8/NSlUBSojQBA X-ME-Proxy: Feedback-ID: idd01497b:Fastmail Received: by mailuser.phl.internal (Postfix, from userid 501) id C34EF780075; Fri, 18 Sep 2026 13:00:32 -0400 (EDT) X-Mailer: MessagingEngine.com Webmail Interface Content-Type: multipart/mixed; boundary=1514ea6d4ce4fd13b31463d1e6e3338e8c33d2e22c7943d9618ebdf08bbc Mime-Version: 1.0 Date: Fri, 18 Sep 2026 17:00:32 +0000 Message-Id: Subject: Stop asserting that Unicode normalization consumes its whole input To: "pgsql-hackers" From: "Tristan Partin" X-Mailer: aerc 0.22.0-22-g3f1cae2c5d5f List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk --1514ea6d4ce4fd13b31463d1e6e3338e8c33d2e22c7943d9618ebdf08bbc Content-Type: multipart/alternative; boundary=2308e736390a46c0ff9ac2b04bc11ce1b19d2fb2c14ef96f0e07f5ef1f1a --2308e736390a46c0ff9ac2b04bc11ce1b19d2fb2c14ef96f0e07f5ef1f1a Content-Transfer-Encoding: quoted-printable Content-Disposition: inline Content-Type: text/plain; charset=UTF-8 normalize() and IS NORMALIZED determine how many code points their input holds with pg_mbstrlen_with_len(). They then decode exactly that many code points, and assert afterward that this consumed the entire datum. That invariant does not hold because pg_mbstrlen_with_len() stops at the first of "limit" or a NUL. Postgres doesn't allow text types to contain NUL, so reaching this requires work, but src/test/regress/sql/encoding.sql already constructs such values to pin down how the string functions behave on them. It just never passed them to normalize() or IS NORMALIZED, which is why this went unnoticed. --=20 Tristan Partin PostgreSQL Contributors Team AWS (https://aws.amazon.com) --2308e736390a46c0ff9ac2b04bc11ce1b19d2fb2c14ef96f0e07f5ef1f1a-- --1514ea6d4ce4fd13b31463d1e6e3338e8c33d2e22c7943d9618ebdf08bbc Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename=v1-0001-Stop-asserting-that-Unicode-normalization-consume.patch Content-Type: text/x-patch; charset=utf-8; name=v1-0001-Stop-asserting-that-Unicode-normalization-consume.patch RnJvbSBkN2E3NGM4MDNhNDJhOWRmMTQzZDdlYjU2MzYxYWE4ZWM2ZjNiOTUxIE1vbiBTZXAgMTcg MDA6MDA6MDAgMjAwMQpGcm9tOiBUcmlzdGFuIFBhcnRpbiA8dHJpc3RhbkBwYXJ0aW4uaW8+CkRh dGU6IEZyaSwgMTEgU2VwIDIwMjYgMDU6Mzc6MTMgKzAwMDAKU3ViamVjdDogW1BBVENIIHYxXSBT dG9wIGFzc2VydGluZyB0aGF0IFVuaWNvZGUgbm9ybWFsaXphdGlvbiBjb25zdW1lcyBpdHMKIHdo b2xlIGlucHV0Cgpub3JtYWxpemUoKSBhbmQgSVMgTk9STUFMSVpFRCBkZXRlcm1pbmUgaG93IG1h bnkgY29kZSBwb2ludHMgdGhlaXIgaW5wdXQKaG9sZHMgd2l0aCBwZ19tYnN0cmxlbl93aXRoX2xl bigpLiBUaGV5IHRoZW4gZGVjb2RlIGV4YWN0bHkgdGhhdCBtYW55CmNvZGUgcG9pbnRzLCBhbmQg YXNzZXJ0IGFmdGVyd2FyZCB0aGF0IHRoaXMgY29uc3VtZWQgdGhlIGVudGlyZSBkYXR1bS4KVGhh dCBpbnZhcmlhbnQgZG9lcyBub3QgaG9sZCBiZWNhdXNlIHBnX21ic3RybGVuX3dpdGhfbGVuKCkg c3RvcHMgYXQgdGhlCmZpcnN0IG9mICJsaW1pdCIgb3IgYSBOVUwuCgpQb3N0Z3JlcyBkb2Vzbid0 IGFsbG93IHRleHQgdHlwZXMgdG8gY29udGFpbiBOVUwsIHNvIHJlYWNoaW5nIHRoaXMKcmVxdWly ZXMgd29yaywgYnV0IHNyYy90ZXN0L3JlZ3Jlc3Mvc3FsL2VuY29kaW5nLnNxbCBhbHJlYWR5IGNv bnN0cnVjdHMKc3VjaCB2YWx1ZXMgdG8gcGluIGRvd24gaG93IHRoZSBzdHJpbmcgZnVuY3Rpb25z IGJlaGF2ZSBvbiB0aGVtLiBJdCBqdXN0Cm5ldmVyIHBhc3NlZCB0aGVtIHRvIG5vcm1hbGl6ZSgp IG9yIElTIE5PUk1BTElaRUQsIHdoaWNoIGlzIHdoeSB0aGlzCndlbnQgdW5ub3RpY2VkLgoKQXV0 aG9yOiBUcmlzdGFuIFBhcnRpbiA8dHJpc3RhbkBwYXJ0aW4uaW8+ClNpZ25lZC1vZmYtYnk6IFRy aXN0YW4gUGFydGluIDx0cmlzdGFuQHBhcnRpbi5pbz4KLS0tCiBzcmMvYmFja2VuZC91dGlscy9h ZHQvdmFybGVuYS5jICAgICAgICB8ICA0ICsrLS0KIHNyYy90ZXN0L3JlZ3Jlc3MvZXhwZWN0ZWQv ZW5jb2Rpbmcub3V0IHwgMTIgKysrKysrKysrKysrCiBzcmMvdGVzdC9yZWdyZXNzL3NxbC9lbmNv ZGluZy5zcWwgICAgICB8ICAyICsrCiAzIGZpbGVzIGNoYW5nZWQsIDE2IGluc2VydGlvbnMoKyks IDIgZGVsZXRpb25zKC0pCgpkaWZmIC0tZ2l0IGEvc3JjL2JhY2tlbmQvdXRpbHMvYWR0L3Zhcmxl bmEuYyBiL3NyYy9iYWNrZW5kL3V0aWxzL2FkdC92YXJsZW5hLmMKaW5kZXggZjZhNDFlNzA5YWUu Ljk2MmI3ODU3YjdkIDEwMDY0NAotLS0gYS9zcmMvYmFja2VuZC91dGlscy9hZHQvdmFybGVuYS5j CisrKyBiL3NyYy9iYWNrZW5kL3V0aWxzL2FkdC92YXJsZW5hLmMKQEAgLTU1MzcsNyArNTUzNyw3 IEBAIHVuaWNvZGVfbm9ybWFsaXplX2Z1bmMoUEdfRlVOQ1RJT05fQVJHUykKIAkJcCArPSBwZ191 dGZfbWJsZW4ocCk7CiAJfQogCWlucHV0X2NoYXJzW2ldID0gKGNoYXIzMl90KSAnXDAnOwotCUFz c2VydCgoY2hhciAqKSBwID09IFZBUkRBVEFfQU5ZKGlucHV0KSArIFZBUlNJWkVfQU5ZX0VYSERS KGlucHV0KSk7CisJQXNzZXJ0KChjaGFyICopIHAgPD0gVkFSREFUQV9BTlkoaW5wdXQpICsgVkFS U0laRV9BTllfRVhIRFIoaW5wdXQpKTsKIAogCS8qIGFjdGlvbiAqLwogCW91dHB1dF9jaGFycyA9 IHVuaWNvZGVfbm9ybWFsaXplKGZvcm0sIGlucHV0X2NoYXJzKTsKQEAgLTU2MDUsNyArNTYwNSw3 IEBAIHVuaWNvZGVfaXNfbm9ybWFsaXplZChQR19GVU5DVElPTl9BUkdTKQogCQlwICs9IHBnX3V0 Zl9tYmxlbihwKTsKIAl9CiAJaW5wdXRfY2hhcnNbaV0gPSAoY2hhcjMyX3QpICdcMCc7Ci0JQXNz ZXJ0KChjaGFyICopIHAgPT0gVkFSREFUQV9BTlkoaW5wdXQpICsgVkFSU0laRV9BTllfRVhIRFIo aW5wdXQpKTsKKwlBc3NlcnQoKGNoYXIgKikgcCA8PSBWQVJEQVRBX0FOWShpbnB1dCkgKyBWQVJT SVpFX0FOWV9FWEhEUihpbnB1dCkpOwogCiAJLyogcXVpY2sgY2hlY2sgKHNlZSBVQVggIzE1KSAq LwogCXF1aWNrY2hlY2sgPSB1bmljb2RlX2lzX25vcm1hbGl6ZWRfcXVpY2tjaGVjayhmb3JtLCBp bnB1dF9jaGFycyk7CmRpZmYgLS1naXQgYS9zcmMvdGVzdC9yZWdyZXNzL2V4cGVjdGVkL2VuY29k aW5nLm91dCBiL3NyYy90ZXN0L3JlZ3Jlc3MvZXhwZWN0ZWQvZW5jb2Rpbmcub3V0CmluZGV4IDBi YjcyYTFkZjZmLi41YzA2ZDZkYWFlNCAxMDA2NDQKLS0tIGEvc3JjL3Rlc3QvcmVncmVzcy9leHBl Y3RlZC9lbmNvZGluZy5vdXQKKysrIGIvc3JjL3Rlc3QvcmVncmVzcy9leHBlY3RlZC9lbmNvZGlu Zy5vdXQKQEAgLTEyMCw2ICsxMjAsMTggQEAgU0VMRUNUIHJlZ2V4cF9yZXBsYWNlKHdpdGhfbnVs LCAnXmNhZiguKSQnLCAnXDEnKSBGUk9NIHJlZ3Jlc3NfZW5jb2Rpbmc7CiAgw6kKICgxIHJvdykK IAorU0VMRUNUIG9jdGV0X2xlbmd0aChub3JtYWxpemUod2l0aF9udWwpKSBGUk9NIHJlZ3Jlc3Nf ZW5jb2Rpbmc7Cisgb2N0ZXRfbGVuZ3RoIAorLS0tLS0tLS0tLS0tLS0KKyAgICAgICAgICAgIDUK KygxIHJvdykKKworU0VMRUNUIHdpdGhfbnVsIElTIE5PUk1BTElaRUQgRlJPTSByZWdyZXNzX2Vu Y29kaW5nOworIGlzX25vcm1hbGl6ZWQgCistLS0tLS0tLS0tLS0tLS0KKyB0CisoMSByb3cpCisK IC0tIE5VTCA9IGNoYXJhY3RlcgogU0VMRUNUIHdpdGhfbnVsLCByZXZlcnNlKHdpdGhfbnVsKSwg cmV2ZXJzZShyZXZlcnNlKHdpdGhfbnVsKSkgRlJPTSByZWdyZXNzX2VuY29kaW5nOwogIHdpdGhf bnVsIHwgcmV2ZXJzZSB8IHJldmVyc2UgCmRpZmYgLS1naXQgYS9zcmMvdGVzdC9yZWdyZXNzL3Nx bC9lbmNvZGluZy5zcWwgYi9zcmMvdGVzdC9yZWdyZXNzL3NxbC9lbmNvZGluZy5zcWwKaW5kZXgg MjZjYWE5M2E1ZDUuLmQyYTY0OWFjYWEyIDEwMDY0NAotLS0gYS9zcmMvdGVzdC9yZWdyZXNzL3Nx bC9lbmNvZGluZy5zcWwKKysrIGIvc3JjL3Rlc3QvcmVncmVzcy9zcWwvZW5jb2Rpbmcuc3FsCkBA IC01Nyw2ICs1Nyw4IEBAIFNFTEVDVCBzdWJzdHJpbmcod2l0aF9udWwsIDQsIDEpIEZST00gcmVn cmVzc19lbmNvZGluZzsKIFNFTEVDVCBzdWJzdHJpbmcod2l0aF9udWwsIDUsIDEpIEZST00gcmVn cmVzc19lbmNvZGluZzsKIFNFTEVDVCBjb252ZXJ0X3RvKHN1YnN0cmluZyh3aXRoX251bCwgNSwg MSksICdVVEY4JykgRlJPTSByZWdyZXNzX2VuY29kaW5nOwogU0VMRUNUIHJlZ2V4cF9yZXBsYWNl KHdpdGhfbnVsLCAnXmNhZiguKSQnLCAnXDEnKSBGUk9NIHJlZ3Jlc3NfZW5jb2Rpbmc7CitTRUxF Q1Qgb2N0ZXRfbGVuZ3RoKG5vcm1hbGl6ZSh3aXRoX251bCkpIEZST00gcmVncmVzc19lbmNvZGlu ZzsKK1NFTEVDVCB3aXRoX251bCBJUyBOT1JNQUxJWkVEIEZST00gcmVncmVzc19lbmNvZGluZzsK IC0tIE5VTCA9IGNoYXJhY3RlcgogU0VMRUNUIHdpdGhfbnVsLCByZXZlcnNlKHdpdGhfbnVsKSwg cmV2ZXJzZShyZXZlcnNlKHdpdGhfbnVsKSkgRlJPTSByZWdyZXNzX2VuY29kaW5nOwogCi0tIApU cmlzdGFuIFBhcnRpbgpodHRwczovL3RyaXN0YW4ucGFydGluLmlvCgo= --1514ea6d4ce4fd13b31463d1e6e3338e8c33d2e22c7943d9618ebdf08bbc--