Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.96) (envelope-from ) id 1wjKKJ-0003Od-0U for pgsql-hackers@arkaria.postgresql.org; Mon, 13 Jul 2026 17:20:19 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.96) (envelope-from ) id 1wjKJH-000TcR-2o for pgsql-hackers@arkaria.postgresql.org; Mon, 13 Jul 2026 17:19:16 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.96) (envelope-from ) id 1wjKJH-000TcI-1t for pgsql-hackers@lists.postgresql.org; Mon, 13 Jul 2026 17:19:16 +0000 Received: from sss.pgh.pa.us ([68.162.161.243]) by magus.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.98.2) (envelope-from ) id 1wjKJG-000000002bw-0pQb for pgsql-hackers@postgresql.org; Mon, 13 Jul 2026 17:19:16 +0000 Received: from sss1.sss.pgh.pa.us (localhost [127.0.0.1]) by sss.pgh.pa.us (8.18.1/8.18.1) with ESMTP id 66DHJ3Ov3228111; Mon, 13 Jul 2026 13:19:03 -0400 From: Tom Lane To: Jeff Davis cc: Chao Li , pgsql-hackers@postgresql.org Subject: Re: Small patch to improve safety of utf8_to_unicode(). In-reply-to: References: <541F240E-94AD-4D65-9794-7D6C316BC3FF@gmail.com> <313BCA0E-2E74-4EB6-9AA8-10DB7F439151@gmail.com> <592441c1d871a4ed0ac1708e132f447caa35869c.camel@j-davis.com> <3058CAA1-40E0-4098-893B-615F77CD35E1@gmail.com> <0cedb517aee8b79b8e31a4e42885ed88c2a67c5f.camel@j-davis.com> <16CAD4DB-9D4A-481A-BB4E-78775C4F4B2C@gmail.com> <3213927.1783950167@sss.pgh.pa.us> Comments: In-reply-to Jeff Davis message dated "Mon, 13 Jul 2026 10:09:34 -0700" MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-ID: <3228109.1783963143.1@sss.pgh.pa.us> Date: Mon, 13 Jul 2026 13:19:03 -0400 Message-ID: <3228110.1783963143@sss.pgh.pa.us> List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk Jeff Davis writes: > On Mon, 2026-07-13 at 09:42 -0400, Tom Lane wrote: >> What I think this is pointing out is that even if the input string's >> length fits in size_t, the length of the case-converted equivalent >> string might not. > From Unicode 17.0 section 5.18.2: "the maximum string expansion as a > result of case mapping in the Unicode Standard is three". Along with a > potential for 4X difference in byte length for a single code point in > UTF8, that makes a maximum of 12X expansion. Well, there aren't going to be any ASCII characters that produce 12 bytes, and probably not any LATIN1 characters either, so I think we could assume that the shortest input character that could produce such a long output would be 3 bytes. So it might be safe to assume that the expansion rate is not more than 4X, which would make this okay (since as you say we can assume the input is < 1GB). Probably needs a little bit of inspection of the LATIN1 cases though, ie could any of those 2-byte characters produce more than 8 bytes? regards, tom lane