pg.ddx.io pgsql-bugs@postgresql.org mailing list archive
help / color / mirror / Atom feedBUG #17277: write past chunk when calling normalize() on an empty string
3+ messages / 2 participants
[nested] [flat]
Commits from this thread:
2021-11-11 098c134556 b609db7155 13c8adf90e de343812b9 56c5a069e0 7653f6f5f5 Fix buffer overrun in unicode string normalization with empty input
* BUG #17277: write past chunk when calling normalize() on an empty string
@ 2021-11-09 21:55 PG Bug reporting form <noreply@postgresql.org>
0 siblings, 1 reply; 3+ messages in thread
From: PG Bug reporting form @ 2021-11-09 21:55 UTC (permalink / raw)
To: pgsql-bugs@lists.postgresql.org; +Cc: postgresql@zr40.nl
The following bug has been logged on the website:
Bug reference: 17277
Logged by: Matthijs van der Vleuten
Email address: postgresql@zr40.nl
PostgreSQL version: 14.0
Operating system: Debian sid
Description:
When calling normalize(''), that is, on an empty string, a warning is
raised: "problem in alloc set ExprContext: detected write past chunk end".
I believe this is due to an error in unicode_norm.c. In unicode_normalize(),
when recompose is true (that is, when using NFC or NFKC normalization) the
loop on line 498 will iterate once before checking count < decomp_size. When
the input is an empty string, this would cause a write outside of the memory
allocated for recomp_chars.
Reproduction:
zr40@[local]:5432 ~=# select version();
version
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────
PostgreSQL 14.0 (Debian 14.0-1.pgdg+1) on x86_64-pc-linux-gnu, compiled by
gcc (Debian 10.3.0-11) 10.3.0, 64-bit
(1 row)
zr40@[local]:5432 ~=# select normalize('');
WARNING: problem in alloc set ExprContext: detected write past chunk end in
block 0x55793d119620, chunk 0x55793d1196a8
WARNING: problem in alloc set ExprContext: detected write past chunk end in
block 0x55793d119620, chunk 0x55793d1196a8
normalize
───────────
(1 row)
^ permalink raw reply [nested|flat] 3+ messages in thread
* Re: BUG #17277: write past chunk when calling normalize() on an empty string
@ 2021-11-10 06:33 Michael Paquier <michael@paquier.xyz>
parent: PG Bug reporting form <noreply@postgresql.org>
0 siblings, 1 reply; 3+ messages in thread
From: Michael Paquier @ 2021-11-10 06:33 UTC (permalink / raw)
To: postgresql@zr40.nl; pgsql-bugs@lists.postgresql.org
On Tue, Nov 09, 2021 at 09:55:08PM +0000, PG Bug reporting form wrote:
> When calling normalize(''), that is, on an empty string, a warning is
> raised: "problem in alloc set ExprContext: detected write past chunk end".
Well, direct callers of unicode_normalize_kc() in ~12 would have the
same problem because this code was not written with this case in mind
as far as I recall, after looking at the git history (60f11b8) as
pg_saslprep() does not allow the case of empty passwords.
> I believe this is due to an error in unicode_norm.c. In unicode_normalize(),
> when recompose is true (that is, when using NFC or NFKC normalization) the
> loop on line 498 will iterate once before checking count < decomp_size. When
> the input is an empty string, this would cause a write outside of the memory
> allocated for recomp_chars.
No, the code does not take the recomposition loop in this case, but
the initialization of target_pos to 1 would cause recomp_chars to be
written past its allocation position by one byte.
As there could be callers of unicode_normalize[_kc]() outside core,
I'd rather fix that at the source and patch unicode_norm.c. One way
to do that would be to leave once you know that there is nothing to
decompose after the loop over decompose_code() and return decomp_chars
that would be set with an empty set of points, as per the attached.
There may be a point in issuing an error if there is an empty string,
though. Another thing would be to consider if is_normalized() should
return false for an empty string, but we have considered empty strings
as normalized since this has been released:
=# SELECT '' IS NFD NORMALIZED;
is_normalized
---------------
t
(1 row)
That feels more natural this way. Still, I can see some perl modules
that would return false for such a case, by the way. The
normalization docs don't seem to mention that directly, except for the
stream-safe text format:
https://www.unicode.org/faq/normalization.html
https://unicode.org/reports/tr15/tr15-51.html
--
Michael
Attachments:
unicode-norm-fix.patch (text/x-diff, 1.7K) download
diff --git a/src/common/unicode_norm.c b/src/common/unicode_norm.c
index 36ff2aab21..06bf921e45 100644
--- a/src/common/unicode_norm.c
+++ b/src/common/unicode_norm.c
@@ -439,6 +439,10 @@ unicode_normalize(UnicodeNormalizationForm form, const pg_wchar *input)
decomp_chars[decomp_size] = '\0';
Assert(decomp_size == current_size);
+ /* Leave if there is nothing to decompose */
+ if (decomp_size == 0)
+ return decomp_chars;
+
/*
* Now apply canonical ordering.
*/
diff --git a/src/test/regress/expected/unicode.out b/src/test/regress/expected/unicode.out
index 2a1e903696..88aef0fe08 100644
--- a/src/test/regress/expected/unicode.out
+++ b/src/test/regress/expected/unicode.out
@@ -8,6 +8,18 @@ SELECT U&'\0061\0308bc' <> U&'\00E4bc' COLLATE "C" AS sanity_check;
t
(1 row)
+SELECT normalize('');
+ normalize
+-----------
+
+(1 row)
+
+SELECT is_normalized('');
+ is_normalized
+---------------
+ t
+(1 row)
+
SELECT normalize(U&'\0061\0308\24D1c') = U&'\00E4\24D1c' COLLATE "C" AS test_default;
test_default
--------------
diff --git a/src/test/regress/sql/unicode.sql b/src/test/regress/sql/unicode.sql
index ccfc6fa77a..07cd2851fc 100644
--- a/src/test/regress/sql/unicode.sql
+++ b/src/test/regress/sql/unicode.sql
@@ -5,6 +5,8 @@ SELECT getdatabaseencoding() <> 'UTF8' AS skip_test \gset
SELECT U&'\0061\0308bc' <> U&'\00E4bc' COLLATE "C" AS sanity_check;
+SELECT normalize('');
+SELECT is_normalized('');
SELECT normalize(U&'\0061\0308\24D1c') = U&'\00E4\24D1c' COLLATE "C" AS test_default;
SELECT normalize(U&'\0061\0308\24D1c', NFC) = U&'\00E4\24D1c' COLLATE "C" AS test_nfc;
SELECT normalize(U&'\00E4bc', NFC) = U&'\00E4bc' COLLATE "C" AS test_nfc_idem;
* Re: BUG #17277: write past chunk when calling normalize() on an empty string
@ 2021-11-12 12:52 Michael Paquier <michael@paquier.xyz>
parent: Michael Paquier <michael@paquier.xyz>
0 siblings, 0 replies; 3+ messages in thread
From: Michael Paquier @ 2021-11-12 12:52 UTC (permalink / raw)
To: postgresql@zr40.nl; pgsql-bugs@lists.postgresql.org
On Wed, Nov 10, 2021 at 03:33:29PM +0900, Michael Paquier wrote:
> That feels more natural this way. Still, I can see some perl modules
> that would return false for such a case, by the way. The
> normalization docs don't seem to mention that directly, except for the
> stream-safe text format:
> https://www.unicode.org/faq/normalization.html
> https://unicode.org/reports/tr15/tr15-51.html
I have expanded the tests, and fixed this one as of 098c1345. Thanks
for the report, Matthijs!
--
Michael
Attachments:
signature.asc (application/pgp-signature, 832B) download
^ permalink raw reply [nested|flat] 3+ messages in thread
end of thread, other threads:[~2021-11-12 12:52 UTC | newest]
Thread overview: 3+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2021-11-09 21:55 BUG #17277: write past chunk when calling normalize() on an empty string PG Bug reporting form <noreply@postgresql.org>
2021-11-10 06:33 ` Michael Paquier <michael@paquier.xyz>
2021-11-12 12:52 ` Michael Paquier <michael@paquier.xyz>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox