From: Michael Paquier <michael@paquier.xyz>
To: postgresql@zr40.nl
To: pgsql-bugs@lists.postgresql.org
Subject: Re: BUG #17277: write past chunk when calling normalize() on an empty string
Date: Wed, 10 Nov 2021 15:33:29 +0900
Message-ID: <YYtnue3sc7EXIIwI@paquier.xyz> (raw)
In-Reply-To: <17277-0c527a373794e802@postgresql.org>
References: <17277-0c527a373794e802@postgresql.org>
On Tue, Nov 09, 2021 at 09:55:08PM +0000, PG Bug reporting form wrote:
> When calling normalize(''), that is, on an empty string, a warning is
> raised: "problem in alloc set ExprContext: detected write past chunk end".
Well, direct callers of unicode_normalize_kc() in ~12 would have the
same problem because this code was not written with this case in mind
as far as I recall, after looking at the git history (60f11b8) as
pg_saslprep() does not allow the case of empty passwords.
> I believe this is due to an error in unicode_norm.c. In unicode_normalize(),
> when recompose is true (that is, when using NFC or NFKC normalization) the
> loop on line 498 will iterate once before checking count < decomp_size. When
> the input is an empty string, this would cause a write outside of the memory
> allocated for recomp_chars.
No, the code does not take the recomposition loop in this case, but
the initialization of target_pos to 1 would cause recomp_chars to be
written past its allocation position by one byte.
As there could be callers of unicode_normalize[_kc]() outside core,
I'd rather fix that at the source and patch unicode_norm.c. One way
to do that would be to leave once you know that there is nothing to
decompose after the loop over decompose_code() and return decomp_chars
that would be set with an empty set of points, as per the attached.
There may be a point in issuing an error if there is an empty string,
though. Another thing would be to consider if is_normalized() should
return false for an empty string, but we have considered empty strings
as normalized since this has been released:
=# SELECT '' IS NFD NORMALIZED;
is_normalized
---------------
t
(1 row)
That feels more natural this way. Still, I can see some perl modules
that would return false for such a case, by the way. The
normalization docs don't seem to mention that directly, except for the
stream-safe text format:
https://www.unicode.org/faq/normalization.htmlhttps://unicode.org/reports/tr15/tr15-51.html
--
Michael
Attachments:unicode-norm-fix.patch (text/x-diff, 1.7K) downloaddiff --git a/src/common/unicode_norm.c b/src/common/unicode_norm.cindex 36ff2aab21..06bf921e45 100644--- a/src/common/unicode_norm.c+++ b/src/common/unicode_norm.c@@ -439,6 +439,10 @@ unicode_normalize(UnicodeNormalizationForm form, const pg_wchar *input)
decomp_chars[decomp_size] = '\0';
Assert(decomp_size == current_size);
+ /* Leave if there is nothing to decompose */+ if (decomp_size == 0)+ return decomp_chars;+
/*
* Now apply canonical ordering.
*/
diff --git a/src/test/regress/expected/unicode.out b/src/test/regress/expected/unicode.outindex 2a1e903696..88aef0fe08 100644--- a/src/test/regress/expected/unicode.out+++ b/src/test/regress/expected/unicode.out@@ -8,6 +8,18 @@ SELECT U&'\0061\0308bc' <> U&'\00E4bc' COLLATE "C" AS sanity_check;
t
(1 row)
+SELECT normalize('');+ normalize +-----------+ +(1 row)++SELECT is_normalized('');+ is_normalized +---------------+ t+(1 row)+
SELECT normalize(U&'\0061\0308\24D1c') = U&'\00E4\24D1c' COLLATE "C" AS test_default;
test_default
--------------
diff --git a/src/test/regress/sql/unicode.sql b/src/test/regress/sql/unicode.sqlindex ccfc6fa77a..07cd2851fc 100644--- a/src/test/regress/sql/unicode.sql+++ b/src/test/regress/sql/unicode.sql@@ -5,6 +5,8 @@ SELECT getdatabaseencoding() <> 'UTF8' AS skip_test \gset
SELECT U&'\0061\0308bc' <> U&'\00E4bc' COLLATE "C" AS sanity_check;
+SELECT normalize('');+SELECT is_normalized('');
SELECT normalize(U&'\0061\0308\24D1c') = U&'\00E4\24D1c' COLLATE "C" AS test_default;
SELECT normalize(U&'\0061\0308\24D1c', NFC) = U&'\00E4\24D1c' COLLATE "C" AS test_nfc;
SELECT normalize(U&'\00E4bc', NFC) = U&'\00E4bc' COLLATE "C" AS test_nfc_idem;
signature.asc (application/pgp-signature, 832B) download
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-bugs@postgresql.org
Cc: michael@paquier.xyz, postgresql@zr40.nl, pgsql-bugs@lists.postgresql.org
Subject: Re: BUG #17277: write past chunk when calling normalize() on an empty string
In-Reply-To: <YYtnue3sc7EXIIwI@paquier.xyz>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox