pg.ddx.io pgsql-bugs@postgresql.org mailing list archive
help / color / mirror / Atom feedBUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation
16+ messages / 4 participants
[nested] [flat]
* BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation
@ 2025-12-02 10:03 PG Bug reporting form <noreply@postgresql.org>
2025-12-02 16:24 ` Re: BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation Laurenz Albe <laurenz.albe@cybertec.at>
0 siblings, 1 reply; 16+ messages in thread
From: PG Bug reporting form @ 2025-12-02 10:03 UTC (permalink / raw)
To: pgsql-bugs@lists.postgresql.org; +Cc: adam.warland@infor.com
The following bug has been logged on the website:
Bug reference: 19341
Logged by: Adam Warland
Email address: adam.warland@infor.com
PostgreSQL version: 18.1
Operating system: Windows 11 Enterprise
Description:
When using a nondeterministic ICU collation, the replace() function fails to
replace a substring when that substring appears at the end of the input
string.
Occurrences of the same substring earlier in the string are replaced
normally.
This appears to be unintended and inconsistent with the documented
limitations of nondeterministic collations. The failure seems specific to
situations where:
• a nondeterministic ICU collation is applied to both source and
match strings, and
• the substring being replaced appears as the final character of the
source string.
The behavior reproduces reliably.
Expected Behavior
replace() should replace all occurrences of the match substring, including
one at the final position, regardless of collation — or, if nondeterministic
collations cannot support this operation, documentation should explicitly
state the limitation (as is already done for LIKE and regular expressions).
Actual Behavior
Under a nondeterministic ICU collation, the final occurrence of the
substring is not replaced, even though earlier occurrences are.
Example output (actual) replace x with y :
res1 | testx -- unchanged, incorrect (final character not replaced)
res2 | yabcdx -- first 'x' replaced, final 'x' not replaced
Under a deterministic or C collation, output is correct:
res_C | testy
Environment
SELECT version();
→ PostgreSQL 18.1 (Debian 18.1-1.pgdg13+2) on x86_64-pc-linux-gnu, compiled
by gcc (Debian 14.2.0-19) 14.2.0, 64-bit.
SHOW server_encoding;
→ UTF8
SHOW lc_collate;
SHOW lc_ctype;
→ en_US.utf8
en_US.utf8
Specific collation used:
create collation test_nondeterministic (
provider = icu,
locale = 'und-u-ks-level2',
deterministic = false
)
Minimal Reproduction Case
drop COLLATION if EXISTS test_nondeterministic;
create collation test_nondeterministic (
provider = icu,
locale = 'und-u-ks-level2',
deterministic = false
);
-- Replace final character under nondeterministic collation
SELECT replace(
'testx' COLLATE "test_nondeterministic",
'x' COLLATE "test_nondeterministic",
'y') AS res1;
-- Replace substring appearing twice — final one fails
SELECT replace(
'xabcdx' COLLATE "test_nondeterministic",
'x' COLLATE "test_nondeterministic",
'y') AS res2;
-- Control test using deterministic collation
SELECT replace(
'testx' COLLATE "C",
'x' COLLATE "C",
'y') AS res_C;
Observed result:
res1 and the final x in res2 are not replaced.
Additional Notes
• The issue does not occur with deterministic ICU collations or with
C collation.
• The failure seems tied specifically to the last character
position, which may indicate an off-by-one issue or a limitation in
substring matching under nondeterministic collation rules.
• No documentation currently states that replace() is partially
unsupported with nondeterministic collations, although other operations
(LIKE, regex) have historically been restricted.
Conclusion
replace() appears to behave incorrectly when matching a substring at the end
of a string under nondeterministic ICU collations. This is either:
• a defect in how nondeterministic collations interact with
substring matching functions, or
an undocumented limitation that should be clarified.
^ permalink raw reply [nested|flat] 16+ messages in thread
* Re: BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation
2025-12-02 10:03 BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation PG Bug reporting form <noreply@postgresql.org>
@ 2025-12-02 16:24 ` Laurenz Albe <laurenz.albe@cybertec.at>
2025-12-02 16:31 ` Re: BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation Laurenz Albe <laurenz.albe@cybertec.at>
2025-12-02 16:36 ` Re: BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation Heikki Linnakangas <hlinnaka@iki.fi>
0 siblings, 2 replies; 16+ messages in thread
From: Laurenz Albe @ 2025-12-02 16:24 UTC (permalink / raw)
To: adam.warland@infor.com; pgsql-bugs@lists.postgresql.org
On Tue, 2025-12-02 at 10:03 +0000, PG Bug reporting form wrote:
> PostgreSQL version: 18.1
>
> When using a nondeterministic ICU collation, the replace() function fails to
> replace a substring when that substring appears at the end of the input
> string.
>
> Occurrences of the same substring earlier in the string are replaced
> normally.
>
> Specific collation used:
> create collation test_nondeterministic (
> provider = icu,
> locale = 'und-u-ks-level2',
> deterministic = false
> )
>
> -- Replace final character under nondeterministic collation
> SELECT replace(
> 'testx' COLLATE "test_nondeterministic",
> 'x' COLLATE "test_nondeterministic",
> 'y') AS res1;
I can reproduce the problem, and the attached patch fixes it for me.
I am not certain if it is safe to apply pg_mblen() to "haystack_end", though.
Yours,
Laurenz Albe
^ permalink raw reply [nested|flat] 16+ messages in thread
* Re: BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation
2025-12-02 10:03 BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation PG Bug reporting form <noreply@postgresql.org>
2025-12-02 16:24 ` Re: BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation Laurenz Albe <laurenz.albe@cybertec.at>
@ 2025-12-02 16:31 ` Laurenz Albe <laurenz.albe@cybertec.at>
1 sibling, 0 replies; 16+ messages in thread
From: Laurenz Albe @ 2025-12-02 16:31 UTC (permalink / raw)
To: adam.warland@infor.com; pgsql-bugs@lists.postgresql.org
On Tue, 2025-12-02 at 17:24 +0100, Laurenz Albe wrote:
> I can reproduce the problem, and the attached patch fixes it for me.
Erm, and here is the patch.
Laurenz Albe
Attachments:
[text/x-patch] v1-0001-Fix-greedy-substring-search-for-non-deterministic.patch (1.3K, ../../a0d41b12dc6067b6e679bfd18f8a050704bc9902.camel@cybertec.at/2-v1-0001-Fix-greedy-substring-search-for-non-deterministic.patch)
download | inline diff:
From 057e4348a2c44816e19e9c5c4711626263a877c4 Mon Sep 17 00:00:00 2001
From: Laurenz Albe <laurenz.albe@cybertec.at>
Date: Tue, 2 Dec 2025 17:20:27 +0100
Subject: [PATCH v1] Fix greedy substring search for non-deterministic
collations
Due to an off-by-one error, the code failed to find matches at the
end of the haystack.
Author: Laurenz Albe <laurenz.albe@cybertec.at>
Reported-By: Adam Warland <adam.warland@infor.com>
Discussion: https://postgr.es/m/19341-1d9a22915edfec58%40postgresql.org
---
src/backend/utils/adt/varlena.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c
index 3894457ab40..7fe109ba73e 100644
--- a/src/backend/utils/adt/varlena.c
+++ b/src/backend/utils/adt/varlena.c
@@ -1149,7 +1149,7 @@ text_position_next_internal(char *start_ptr, TextPositionState *state)
* Else check if any of the possible substrings starting at hptr
* are equal to the needle.
*/
- for (const char *test_end = hptr; test_end < haystack_end; test_end += pg_mblen(test_end))
+ for (const char *test_end = hptr; test_end <= haystack_end; test_end += pg_mblen(test_end))
{
if (pg_strncoll(hptr, (test_end - hptr), needle, needle_len, state->locale) == 0)
{
--
2.52.0
^ permalink raw reply [nested|flat] 16+ messages in thread
* Re: BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation
2025-12-02 10:03 BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation PG Bug reporting form <noreply@postgresql.org>
2025-12-02 16:24 ` Re: BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation Laurenz Albe <laurenz.albe@cybertec.at>
@ 2025-12-02 16:36 ` Heikki Linnakangas <hlinnaka@iki.fi>
2025-12-02 17:18 ` Re: BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation Laurenz Albe <laurenz.albe@cybertec.at>
2025-12-02 17:29 ` Re: BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation Heikki Linnakangas <hlinnaka@iki.fi>
1 sibling, 2 replies; 16+ messages in thread
From: Heikki Linnakangas @ 2025-12-02 16:36 UTC (permalink / raw)
To: Laurenz Albe <laurenz.albe@cybertec.at>; adam.warland@infor.com; pgsql-bugs@lists.postgresql.org
On 02/12/2025 18:24, Laurenz Albe wrote:
> On Tue, 2025-12-02 at 10:03 +0000, PG Bug reporting form wrote:
>> PostgreSQL version: 18.1
>>
>> When using a nondeterministic ICU collation, the replace() function fails to
>> replace a substring when that substring appears at the end of the input
>> string.
>>
>> Occurrences of the same substring earlier in the string are replaced
>> normally.
>>
>> Specific collation used:
>> create collation test_nondeterministic (
>> provider = icu,
>> locale = 'und-u-ks-level2',
>> deterministic = false
>> )
>>
>> -- Replace final character under nondeterministic collation
>> SELECT replace(
>> 'testx' COLLATE "test_nondeterministic",
>> 'x' COLLATE "test_nondeterministic",
>> 'y') AS res1;
>
> I can reproduce the problem, and the attached patch fixes it for me.
+1, looks good to me. Let's also add a regression test for this.
> I am not certain if it is safe to apply pg_mblen() to "haystack_end", though.
It doesn't do that though, does it? There are two pg_mblen() calls in
the vicinity:
> for (const char *test_end = hptr; test_end <= haystack_end; test_end += pg_mblen(test_end))
> {
> if (pg_strncoll(hptr, (test_end - hptr), needle, needle_len, state->locale) == 0)
> {
> state->last_match_len_tmp = (test_end - hptr);
> result_hptr = hptr;
> if (!state->greedy)
> break;
> }
> }
> if (result_hptr)
> break;
>
> hptr += pg_mblen(hptr);
Neither of those will get called with 'haystack_end' as far as I can see.
- Heikki
^ permalink raw reply [nested|flat] 16+ messages in thread
* Re: BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation
2025-12-02 10:03 BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation PG Bug reporting form <noreply@postgresql.org>
2025-12-02 16:24 ` Re: BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation Laurenz Albe <laurenz.albe@cybertec.at>
2025-12-02 16:36 ` Re: BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation Heikki Linnakangas <hlinnaka@iki.fi>
@ 2025-12-02 17:18 ` Laurenz Albe <laurenz.albe@cybertec.at>
2025-12-02 17:25 ` Re: BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation Tom Lane <tgl@sss.pgh.pa.us>
1 sibling, 1 reply; 16+ messages in thread
From: Laurenz Albe @ 2025-12-02 17:18 UTC (permalink / raw)
To: Heikki Linnakangas <hlinnaka@iki.fi>; adam.warland@infor.com; pgsql-bugs@lists.postgresql.org
On Tue, 2025-12-02 at 18:36 +0200, Heikki Linnakangas wrote:
> +1, looks good to me. Let's also add a regression test for this.
Right, done in the attached.
> > I am not certain if it is safe to apply pg_mblen() to "haystack_end", though.
>
> It doesn't do that though, does it? There are two pg_mblen() calls in
> the vicinity:
>
> > for (const char *test_end = hptr; test_end <= haystack_end; test_end += pg_mblen(test_end))
> > {
> > if (pg_strncoll(hptr, (test_end - hptr), needle, needle_len, state->locale) == 0)
> > {
> > state->last_match_len_tmp = (test_end - hptr);
> > result_hptr = hptr;
> > if (!state->greedy)
> > break;
> > }
> > }
> > if (result_hptr)
> > break;
> >
> > hptr += pg_mblen(hptr);
>
> Neither of those will get called with 'haystack_end' as far as I can see.
During the last iteration of the loop, "test_end" will be equal to "haystack_end",
and the loop increment will call "pg_mblen(test_end)".
Yours,
Laurenz Albe
Attachments:
[text/x-patch] v2-0001-Fix-greedy-substring-search-for-non-deterministic.patch (2.9K, ../../9f47c6b24d6cf59002603caf19b8b3f4854176be.camel@cybertec.at/2-v2-0001-Fix-greedy-substring-search-for-non-deterministic.patch)
download | inline diff:
From f547c38f7fffcaa4c744aea22e16ca9194505916 Mon Sep 17 00:00:00 2001
From: Laurenz Albe <laurenz.albe@cybertec.at>
Date: Tue, 2 Dec 2025 18:15:09 +0100
Subject: [PATCH v2] Fix greedy substring search for non-deterministic
collations
Due to an off-by-one error, the code failed to find matches at the
end of the haystack.
Author: Laurenz Albe <laurenz.albe@cybertec.at>
Reported-By: Adam Warland <adam.warland@infor.com>
Discussion: https://postgr.es/m/19341-1d9a22915edfec58%40postgresql.org
---
src/backend/utils/adt/varlena.c | 2 +-
src/test/regress/expected/collate.icu.utf8.out | 7 +++++++
src/test/regress/sql/collate.icu.utf8.sql | 3 +++
3 files changed, 11 insertions(+), 1 deletion(-)
diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c
index 3894457ab40..7fe109ba73e 100644
--- a/src/backend/utils/adt/varlena.c
+++ b/src/backend/utils/adt/varlena.c
@@ -1149,7 +1149,7 @@ text_position_next_internal(char *start_ptr, TextPositionState *state)
* Else check if any of the possible substrings starting at hptr
* are equal to the needle.
*/
- for (const char *test_end = hptr; test_end < haystack_end; test_end += pg_mblen(test_end))
+ for (const char *test_end = hptr; test_end <= haystack_end; test_end += pg_mblen(test_end))
{
if (pg_strncoll(hptr, (test_end - hptr), needle, needle_len, state->locale) == 0)
{
diff --git a/src/test/regress/expected/collate.icu.utf8.out b/src/test/regress/expected/collate.icu.utf8.out
index b8579a1efc6..0a14c1d93ff 100644
--- a/src/test/regress/expected/collate.icu.utf8.out
+++ b/src/test/regress/expected/collate.icu.utf8.out
@@ -1484,6 +1484,13 @@ SELECT array_sort('{a,B}'::text[] COLLATE "C");
{B,a}
(1 row)
+-- test replace() at the end of the string (bug #19341)
+SELECT replace('testX' COLLATE case_insensitive, 'x' COLLATE case_insensitive, 'es');
+ replace
+---------
+ testes
+(1 row)
+
-- test language tags
CREATE COLLATION lt_insensitive (provider = icu, locale = 'en-u-ks-level1', deterministic = false);
SELECT 'aBcD' COLLATE lt_insensitive = 'AbCd' COLLATE lt_insensitive;
diff --git a/src/test/regress/sql/collate.icu.utf8.sql b/src/test/regress/sql/collate.icu.utf8.sql
index 6f5abac0dc0..5e3fef9b381 100644
--- a/src/test/regress/sql/collate.icu.utf8.sql
+++ b/src/test/regress/sql/collate.icu.utf8.sql
@@ -568,6 +568,9 @@ SELECT 'abc' <= 'ABC' COLLATE case_insensitive, 'abc' >= 'ABC' COLLATE case_inse
SELECT array_sort('{a,B}'::text[] COLLATE case_insensitive);
SELECT array_sort('{a,B}'::text[] COLLATE "C");
+-- test replace() at the end of the string (bug #19341)
+SELECT replace('testX' COLLATE case_insensitive, 'x' COLLATE case_insensitive, 'es');
+
-- test language tags
CREATE COLLATION lt_insensitive (provider = icu, locale = 'en-u-ks-level1', deterministic = false);
SELECT 'aBcD' COLLATE lt_insensitive = 'AbCd' COLLATE lt_insensitive;
--
2.52.0
^ permalink raw reply [nested|flat] 16+ messages in thread
* Re: BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation
2025-12-02 10:03 BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation PG Bug reporting form <noreply@postgresql.org>
2025-12-02 16:24 ` Re: BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation Laurenz Albe <laurenz.albe@cybertec.at>
2025-12-02 16:36 ` Re: BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation Heikki Linnakangas <hlinnaka@iki.fi>
2025-12-02 17:18 ` Re: BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation Laurenz Albe <laurenz.albe@cybertec.at>
@ 2025-12-02 17:25 ` Tom Lane <tgl@sss.pgh.pa.us>
2025-12-02 17:51 ` Re: BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation Laurenz Albe <laurenz.albe@cybertec.at>
0 siblings, 1 reply; 16+ messages in thread
From: Tom Lane @ 2025-12-02 17:25 UTC (permalink / raw)
To: Laurenz Albe <laurenz.albe@cybertec.at>; +Cc: Heikki Linnakangas <hlinnaka@iki.fi>; adam.warland@infor.com; pgsql-bugs@lists.postgresql.org
Laurenz Albe <laurenz.albe@cybertec.at> writes:
>>> for (const char *test_end = hptr; test_end <= haystack_end; test_end += pg_mblen(test_end))
> During the last iteration of the loop, "test_end" will be equal to "haystack_end",
> and the loop increment will call "pg_mblen(test_end)".
Right, clearly unsafe (and I bet valgrind would complain about it).
You need to rearrange the loop logic so that we won't attempt to
increment test_end that last time through. Perhaps a for-loop
isn't the best way to write it.
regards, tom lane
^ permalink raw reply [nested|flat] 16+ messages in thread
* Re: BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation
2025-12-02 10:03 BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation PG Bug reporting form <noreply@postgresql.org>
2025-12-02 16:24 ` Re: BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation Laurenz Albe <laurenz.albe@cybertec.at>
2025-12-02 16:36 ` Re: BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation Heikki Linnakangas <hlinnaka@iki.fi>
2025-12-02 17:18 ` Re: BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation Laurenz Albe <laurenz.albe@cybertec.at>
2025-12-02 17:25 ` Re: BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation Tom Lane <tgl@sss.pgh.pa.us>
@ 2025-12-02 17:51 ` Laurenz Albe <laurenz.albe@cybertec.at>
2025-12-02 20:53 ` Re: BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 1 reply; 16+ messages in thread
From: Laurenz Albe @ 2025-12-02 17:51 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Heikki Linnakangas <hlinnaka@iki.fi>; adam.warland@infor.com; pgsql-bugs@lists.postgresql.org
On Tue, 2025-12-02 at 12:25 -0500, Tom Lane wrote:
> Laurenz Albe <laurenz.albe@cybertec.at> writes:
> > > > for (const char *test_end = hptr; test_end <= haystack_end; test_end += pg_mblen(test_end))
>
> > During the last iteration of the loop, "test_end" will be equal to "haystack_end",
> > and the loop increment will call "pg_mblen(test_end)".
>
> Right, clearly unsafe (and I bet valgrind would complain about it).
> You need to rearrange the loop logic so that we won't attempt to
> increment test_end that last time through. Perhaps a for-loop
> isn't the best way to write it.
Right. The attached patch v3 turns it into a while loop to avoid
the problem.
Yours,
Laurenz Albe
Attachments:
[text/x-patch] v3-0001-Fix-greedy-substring-search-for-non-deterministic.patch (3.5K, ../../464a14fcbee9567ace136716714b8c18f9bd87fb.camel@cybertec.at/2-v3-0001-Fix-greedy-substring-search-for-non-deterministic.patch)
download | inline diff:
From 41c441a02db341952763d04bb822a634b711c735 Mon Sep 17 00:00:00 2001
From: Laurenz Albe <laurenz.albe@cybertec.at>
Date: Tue, 2 Dec 2025 18:49:11 +0100
Subject: [PATCH v3] Fix greedy substring search for non-deterministic
collations
Due to an off-by-one error, the code failed to find matches at the
end of the haystack.
Author: Laurenz Albe <laurenz.albe@cybertec.at>
Reported-By: Adam Warland <adam.warland@infor.com>
Reviewed-By: Heikki Linnakangas <hlinnaka@iki.fi>
Reviewed-By: Tom Lane <tgl@sss.pgh.pa.us>
Discussion: https://postgr.es/m/19341-1d9a22915edfec58%40postgresql.org
---
src/backend/utils/adt/varlena.c | 9 ++++++++-
src/test/regress/expected/collate.icu.utf8.out | 7 +++++++
src/test/regress/sql/collate.icu.utf8.sql | 3 +++
3 files changed, 18 insertions(+), 1 deletion(-)
diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c
index 3894457ab40..99d1bf09f20 100644
--- a/src/backend/utils/adt/varlena.c
+++ b/src/backend/utils/adt/varlena.c
@@ -1136,6 +1136,8 @@ text_position_next_internal(char *start_ptr, TextPositionState *state)
hptr = start_ptr;
while (hptr < haystack_end)
{
+ const char *test_end;
+
/*
* First check the common case that there is a match in the
* haystack of exactly the length of the needle.
@@ -1149,7 +1151,8 @@ text_position_next_internal(char *start_ptr, TextPositionState *state)
* Else check if any of the possible substrings starting at hptr
* are equal to the needle.
*/
- for (const char *test_end = hptr; test_end < haystack_end; test_end += pg_mblen(test_end))
+ test_end = hptr;
+ while (test_end <= haystack_end)
{
if (pg_strncoll(hptr, (test_end - hptr), needle, needle_len, state->locale) == 0)
{
@@ -1158,6 +1161,10 @@ text_position_next_internal(char *start_ptr, TextPositionState *state)
if (!state->greedy)
break;
}
+
+ if (test_end == haystack_end)
+ break;
+ test_end += pg_mblen(test_end);
}
if (result_hptr)
break;
diff --git a/src/test/regress/expected/collate.icu.utf8.out b/src/test/regress/expected/collate.icu.utf8.out
index b8579a1efc6..0a14c1d93ff 100644
--- a/src/test/regress/expected/collate.icu.utf8.out
+++ b/src/test/regress/expected/collate.icu.utf8.out
@@ -1484,6 +1484,13 @@ SELECT array_sort('{a,B}'::text[] COLLATE "C");
{B,a}
(1 row)
+-- test replace() at the end of the string (bug #19341)
+SELECT replace('testX' COLLATE case_insensitive, 'x' COLLATE case_insensitive, 'es');
+ replace
+---------
+ testes
+(1 row)
+
-- test language tags
CREATE COLLATION lt_insensitive (provider = icu, locale = 'en-u-ks-level1', deterministic = false);
SELECT 'aBcD' COLLATE lt_insensitive = 'AbCd' COLLATE lt_insensitive;
diff --git a/src/test/regress/sql/collate.icu.utf8.sql b/src/test/regress/sql/collate.icu.utf8.sql
index 6f5abac0dc0..5e3fef9b381 100644
--- a/src/test/regress/sql/collate.icu.utf8.sql
+++ b/src/test/regress/sql/collate.icu.utf8.sql
@@ -568,6 +568,9 @@ SELECT 'abc' <= 'ABC' COLLATE case_insensitive, 'abc' >= 'ABC' COLLATE case_inse
SELECT array_sort('{a,B}'::text[] COLLATE case_insensitive);
SELECT array_sort('{a,B}'::text[] COLLATE "C");
+-- test replace() at the end of the string (bug #19341)
+SELECT replace('testX' COLLATE case_insensitive, 'x' COLLATE case_insensitive, 'es');
+
-- test language tags
CREATE COLLATION lt_insensitive (provider = icu, locale = 'en-u-ks-level1', deterministic = false);
SELECT 'aBcD' COLLATE lt_insensitive = 'AbCd' COLLATE lt_insensitive;
--
2.52.0
^ permalink raw reply [nested|flat] 16+ messages in thread
* Re: BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation
2025-12-02 10:03 BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation PG Bug reporting form <noreply@postgresql.org>
2025-12-02 16:24 ` Re: BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation Laurenz Albe <laurenz.albe@cybertec.at>
2025-12-02 16:36 ` Re: BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation Heikki Linnakangas <hlinnaka@iki.fi>
2025-12-02 17:18 ` Re: BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation Laurenz Albe <laurenz.albe@cybertec.at>
2025-12-02 17:25 ` Re: BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation Tom Lane <tgl@sss.pgh.pa.us>
2025-12-02 17:51 ` Re: BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation Laurenz Albe <laurenz.albe@cybertec.at>
@ 2025-12-02 20:53 ` Tom Lane <tgl@sss.pgh.pa.us>
2025-12-02 21:44 ` Re: BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation Laurenz Albe <laurenz.albe@cybertec.at>
2025-12-03 07:51 ` Re: BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation Laurenz Albe <laurenz.albe@cybertec.at>
0 siblings, 2 replies; 16+ messages in thread
From: Tom Lane @ 2025-12-02 20:53 UTC (permalink / raw)
To: Laurenz Albe <laurenz.albe@cybertec.at>; +Cc: Heikki Linnakangas <hlinnaka@iki.fi>; adam.warland@infor.com; pgsql-bugs@lists.postgresql.org
Laurenz Albe <laurenz.albe@cybertec.at> writes:
> On Tue, 2025-12-02 at 12:25 -0500, Tom Lane wrote:
>> You need to rearrange the loop logic so that we won't attempt to
>> increment test_end that last time through. Perhaps a for-loop
>> isn't the best way to write it.
> Right. The attached patch v3 turns it into a while loop to avoid
> the problem.
Looking at the code overall, I wonder if the outer loop doesn't have
the same issue. The comments claim that we should be able to handle
zero-length matches, but if the overall haystack is of length zero,
we will fail to check for such a match.
Also, since we have haystack <= haystack_end as a starting condition,
I think both loops could omit the initial test. I'd be inclined
to code them like
test_ptr = start point;
for (;;)
{
...
if (test_ptr >= haystack_end)
break;
test_ptr += pg_mblen(test_ptr);
}
On the other hand ... is that comment really right about zero-length
match being possible? If it is, the API for this function is in
need of redesign, because callers that try to find "the next match"
would go into an infinite loop re-finding the same zero-length
match over and over.
regards, tom lane
^ permalink raw reply [nested|flat] 16+ messages in thread
* Re: BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation
2025-12-02 10:03 BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation PG Bug reporting form <noreply@postgresql.org>
2025-12-02 16:24 ` Re: BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation Laurenz Albe <laurenz.albe@cybertec.at>
2025-12-02 16:36 ` Re: BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation Heikki Linnakangas <hlinnaka@iki.fi>
2025-12-02 17:18 ` Re: BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation Laurenz Albe <laurenz.albe@cybertec.at>
2025-12-02 17:25 ` Re: BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation Tom Lane <tgl@sss.pgh.pa.us>
2025-12-02 17:51 ` Re: BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation Laurenz Albe <laurenz.albe@cybertec.at>
2025-12-02 20:53 ` Re: BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation Tom Lane <tgl@sss.pgh.pa.us>
@ 2025-12-02 21:44 ` Laurenz Albe <laurenz.albe@cybertec.at>
1 sibling, 0 replies; 16+ messages in thread
From: Laurenz Albe @ 2025-12-02 21:44 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Heikki Linnakangas <hlinnaka@iki.fi>; adam.warland@infor.com; pgsql-bugs@lists.postgresql.org; peter@eisentraut.org
On Tue, 2025-12-02 at 15:53 -0500, Tom Lane wrote:
> On the other hand ... is that comment really right about zero-length
> match being possible? If it is, the API for this function is in
> need of redesign, because callers that try to find "the next match"
> would go into an infinite loop re-finding the same zero-length
> match over and over.
I know too little about exotic collations to answer that, but it sure
would be a problem. All I find in the discussion is the claim by
Peter E. in [1] that it is so. Perhaps he can enlighten us.
Yours,
Laurenz Albe
[1]: https://postgr.es/m/6107daa2-5cf7-4cf2-a526-626be1d15b18%40eisentraut.org
^ permalink raw reply [nested|flat] 16+ messages in thread
* Re: BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation
2025-12-02 10:03 BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation PG Bug reporting form <noreply@postgresql.org>
2025-12-02 16:24 ` Re: BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation Laurenz Albe <laurenz.albe@cybertec.at>
2025-12-02 16:36 ` Re: BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation Heikki Linnakangas <hlinnaka@iki.fi>
2025-12-02 17:18 ` Re: BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation Laurenz Albe <laurenz.albe@cybertec.at>
2025-12-02 17:25 ` Re: BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation Tom Lane <tgl@sss.pgh.pa.us>
2025-12-02 17:51 ` Re: BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation Laurenz Albe <laurenz.albe@cybertec.at>
2025-12-02 20:53 ` Re: BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation Tom Lane <tgl@sss.pgh.pa.us>
@ 2025-12-03 07:51 ` Laurenz Albe <laurenz.albe@cybertec.at>
2025-12-03 15:12 ` Re: BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation Tom Lane <tgl@sss.pgh.pa.us>
1 sibling, 1 reply; 16+ messages in thread
From: Laurenz Albe @ 2025-12-03 07:51 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Heikki Linnakangas <hlinnaka@iki.fi>; adam.warland@infor.com; pgsql-bugs@lists.postgresql.org
On Tue, 2025-12-02 at 15:53 -0500, Tom Lane wrote:
> > The attached patch v3 turns it into a while loop to avoid
> > the problem.
>
> Looking at the code overall, I wonder if the outer loop doesn't have
> the same issue. The comments claim that we should be able to handle
> zero-length matches, but if the overall haystack is of length zero,
> we will fail to check for such a match.
If you can find zero-length matches at all, you could find a
zero-length match in a non-empty haystack. Perhaps the function is
never called with an empty haystack...
> Also, since we have haystack <= haystack_end as a starting condition,
> I think both loops could omit the initial test. I'd be inclined
> to code them like
>
> test_ptr = start point;
> for (;;)
> {
> ...
> if (test_ptr >= haystack_end)
> break;
> test_ptr += pg_mblen(test_ptr);
> }
True. The attached v4 patch does it like that.
> On the other hand ... is that comment really right about zero-length
> match being possible? If it is, the API for this function is in
> need of redesign, because callers that try to find "the next match"
> would go into an infinite loop re-finding the same zero-length
> match over and over.
Right. I'll see if I can trigger such a case.
Yours,
Laurenz Albe
Attachments:
[text/x-patch] v4-0001-Fix-greedy-substring-search-for-non-deterministic.patch (3.5K, ../../0bbe14779f2c37e5a2fbd514861ebedaf8f53df6.camel@cybertec.at/2-v4-0001-Fix-greedy-substring-search-for-non-deterministic.patch)
download | inline diff:
From 8a4363a1a6222a03c40e3df413faa84f810252ee Mon Sep 17 00:00:00 2001
From: Laurenz Albe <laurenz.albe@cybertec.at>
Date: Wed, 3 Dec 2025 08:41:45 +0100
Subject: [PATCH v4] Fix greedy substring search for non-deterministic
collations
Due to an off-by-one error, the code failed to find matches at the
end of the haystack.
Author: Laurenz Albe <laurenz.albe@cybertec.at>
Reported-By: Adam Warland <adam.warland@infor.com>
Reviewed-By: Heikki Linnakangas <hlinnaka@iki.fi>
Reviewed-By: Tom Lane <tgl@sss.pgh.pa.us>
Discussion: https://postgr.es/m/19341-1d9a22915edfec58%40postgresql.org
---
src/backend/utils/adt/varlena.c | 9 ++++++++-
src/test/regress/expected/collate.icu.utf8.out | 7 +++++++
src/test/regress/sql/collate.icu.utf8.sql | 3 +++
3 files changed, 18 insertions(+), 1 deletion(-)
diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c
index 3894457ab40..d4d0b3531af 100644
--- a/src/backend/utils/adt/varlena.c
+++ b/src/backend/utils/adt/varlena.c
@@ -1136,6 +1136,8 @@ text_position_next_internal(char *start_ptr, TextPositionState *state)
hptr = start_ptr;
while (hptr < haystack_end)
{
+ const char *test_end;
+
/*
* First check the common case that there is a match in the
* haystack of exactly the length of the needle.
@@ -1149,7 +1151,8 @@ text_position_next_internal(char *start_ptr, TextPositionState *state)
* Else check if any of the possible substrings starting at hptr
* are equal to the needle.
*/
- for (const char *test_end = hptr; test_end < haystack_end; test_end += pg_mblen(test_end))
+ test_end = hptr;
+ for (;;)
{
if (pg_strncoll(hptr, (test_end - hptr), needle, needle_len, state->locale) == 0)
{
@@ -1158,6 +1161,10 @@ text_position_next_internal(char *start_ptr, TextPositionState *state)
if (!state->greedy)
break;
}
+
+ if (test_end >= haystack_end)
+ break;
+ test_end += pg_mblen(test_end);
}
if (result_hptr)
break;
diff --git a/src/test/regress/expected/collate.icu.utf8.out b/src/test/regress/expected/collate.icu.utf8.out
index b8579a1efc6..0a14c1d93ff 100644
--- a/src/test/regress/expected/collate.icu.utf8.out
+++ b/src/test/regress/expected/collate.icu.utf8.out
@@ -1484,6 +1484,13 @@ SELECT array_sort('{a,B}'::text[] COLLATE "C");
{B,a}
(1 row)
+-- test replace() at the end of the string (bug #19341)
+SELECT replace('testX' COLLATE case_insensitive, 'x' COLLATE case_insensitive, 'es');
+ replace
+---------
+ testes
+(1 row)
+
-- test language tags
CREATE COLLATION lt_insensitive (provider = icu, locale = 'en-u-ks-level1', deterministic = false);
SELECT 'aBcD' COLLATE lt_insensitive = 'AbCd' COLLATE lt_insensitive;
diff --git a/src/test/regress/sql/collate.icu.utf8.sql b/src/test/regress/sql/collate.icu.utf8.sql
index 6f5abac0dc0..5e3fef9b381 100644
--- a/src/test/regress/sql/collate.icu.utf8.sql
+++ b/src/test/regress/sql/collate.icu.utf8.sql
@@ -568,6 +568,9 @@ SELECT 'abc' <= 'ABC' COLLATE case_insensitive, 'abc' >= 'ABC' COLLATE case_inse
SELECT array_sort('{a,B}'::text[] COLLATE case_insensitive);
SELECT array_sort('{a,B}'::text[] COLLATE "C");
+-- test replace() at the end of the string (bug #19341)
+SELECT replace('testX' COLLATE case_insensitive, 'x' COLLATE case_insensitive, 'es');
+
-- test language tags
CREATE COLLATION lt_insensitive (provider = icu, locale = 'en-u-ks-level1', deterministic = false);
SELECT 'aBcD' COLLATE lt_insensitive = 'AbCd' COLLATE lt_insensitive;
--
2.52.0
^ permalink raw reply [nested|flat] 16+ messages in thread
* Re: BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation
2025-12-02 10:03 BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation PG Bug reporting form <noreply@postgresql.org>
2025-12-02 16:24 ` Re: BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation Laurenz Albe <laurenz.albe@cybertec.at>
2025-12-02 16:36 ` Re: BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation Heikki Linnakangas <hlinnaka@iki.fi>
2025-12-02 17:18 ` Re: BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation Laurenz Albe <laurenz.albe@cybertec.at>
2025-12-02 17:25 ` Re: BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation Tom Lane <tgl@sss.pgh.pa.us>
2025-12-02 17:51 ` Re: BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation Laurenz Albe <laurenz.albe@cybertec.at>
2025-12-02 20:53 ` Re: BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation Tom Lane <tgl@sss.pgh.pa.us>
2025-12-03 07:51 ` Re: BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation Laurenz Albe <laurenz.albe@cybertec.at>
@ 2025-12-03 15:12 ` Tom Lane <tgl@sss.pgh.pa.us>
2025-12-04 17:14 ` Re: BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation Laurenz Albe <laurenz.albe@cybertec.at>
0 siblings, 1 reply; 16+ messages in thread
From: Tom Lane @ 2025-12-03 15:12 UTC (permalink / raw)
To: Laurenz Albe <laurenz.albe@cybertec.at>; +Cc: Heikki Linnakangas <hlinnaka@iki.fi>; adam.warland@infor.com; pgsql-bugs@lists.postgresql.org
Laurenz Albe <laurenz.albe@cybertec.at> writes:
> On Tue, 2025-12-02 at 15:53 -0500, Tom Lane wrote:
>> Looking at the code overall, I wonder if the outer loop doesn't have
>> the same issue. The comments claim that we should be able to handle
>> zero-length matches, but if the overall haystack is of length zero,
>> we will fail to check for such a match.
> If you can find zero-length matches at all, you could find a
> zero-length match in a non-empty haystack. Perhaps the function is
> never called with an empty haystack...
After further thought, it seems to me that this comment is an
unjustified extrapolation from what Peter actually said, which was
that the match substring could be physically shorter than the needle.
Which is certainly true, for instance case-folding or accent-stripping
might shorten the string. But it doesn't follow that a nonempty
needle could ever match an empty substring; and that does not seem
like it could be sane behavior to me. We're considering string
comparison here, not regexes.
We do require callers to eliminate the empty-needle case, and given
that I think we could assume that match substrings must be at least
1 byte long. That assumption is what justifies the current API for
these functions, and perhaps we can also simplify this loop by
using it.
regards, tom lane
^ permalink raw reply [nested|flat] 16+ messages in thread
* Re: BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation
2025-12-02 10:03 BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation PG Bug reporting form <noreply@postgresql.org>
2025-12-02 16:24 ` Re: BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation Laurenz Albe <laurenz.albe@cybertec.at>
2025-12-02 16:36 ` Re: BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation Heikki Linnakangas <hlinnaka@iki.fi>
2025-12-02 17:18 ` Re: BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation Laurenz Albe <laurenz.albe@cybertec.at>
2025-12-02 17:25 ` Re: BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation Tom Lane <tgl@sss.pgh.pa.us>
2025-12-02 17:51 ` Re: BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation Laurenz Albe <laurenz.albe@cybertec.at>
2025-12-02 20:53 ` Re: BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation Tom Lane <tgl@sss.pgh.pa.us>
2025-12-03 07:51 ` Re: BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation Laurenz Albe <laurenz.albe@cybertec.at>
2025-12-03 15:12 ` Re: BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation Tom Lane <tgl@sss.pgh.pa.us>
@ 2025-12-04 17:14 ` Laurenz Albe <laurenz.albe@cybertec.at>
2025-12-06 01:12 ` Re: BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 1 reply; 16+ messages in thread
From: Laurenz Albe @ 2025-12-04 17:14 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Heikki Linnakangas <hlinnaka@iki.fi>; adam.warland@infor.com; pgsql-bugs@lists.postgresql.org
On Wed, 2025-12-03 at 10:12 -0500, Tom Lane wrote:
> Laurenz Albe <laurenz.albe@cybertec.at> writes:
> > On Tue, 2025-12-02 at 15:53 -0500, Tom Lane wrote:
> > > Looking at the code overall, I wonder if the outer loop doesn't have
> > > the same issue. The comments claim that we should be able to handle
> > > zero-length matches, but if the overall haystack is of length zero,
> > > we will fail to check for such a match.
>
>
> After further thought, it seems to me that this comment is an
> unjustified extrapolation from what Peter actually said, which was
> that the match substring could be physically shorter than the needle.
> Which is certainly true, for instance case-folding or accent-stripping
> might shorten the string. But it doesn't follow that a nonempty
> needle could ever match an empty substring; and that does not seem
> like it could be sane behavior to me. We're considering string
> comparison here, not regexes.
>
> We do require callers to eliminate the empty-needle case, and given
> that I think we could assume that match substrings must be at least
> 1 byte long. That assumption is what justifies the current API for
> these functions, and perhaps we can also simplify this loop by
> using it.
I think I get it. I don't see an explicit requirement for a non-empty
needle, but all callers of text_position_next_internal() handle that
case separately.
The attached v5 patch simplifies the loop to a do-while loop, assuming
that we cannot find a zero-length match.
I have also updated the comments to no longer mention the possibility
of an empty match, and for good measure I have added an Assert() that
the needle cannot be empty.
Yours,
Laurenz Albe
Attachments:
[text/x-patch] v5-0001-Fix-greedy-substring-search-for-non-deterministic.patch (5.1K, ../../792a0aeb486e240ea34f10f895e0368fc434b8b7.camel@cybertec.at/2-v5-0001-Fix-greedy-substring-search-for-non-deterministic.patch)
download | inline diff:
From e265fc5790d4c8224bb758518470b2424418d913 Mon Sep 17 00:00:00 2001
From: Laurenz Albe <laurenz.albe@cybertec.at>
Date: Thu, 4 Dec 2025 18:07:06 +0100
Subject: [PATCH v5] Fix greedy substring search for non-deterministic
collations
Due to an off-by-one error, the code failed to find matches at the
end of the haystack. Fix by rewriting the loop.
While at it, fix a comment that claimed that the function could find
a zero-length match. Such a match could send a caller into an
endless loop. However, zero-length matches can only occur with an
empty search string, and that case is explicitly excluded by all
callers. To make sure it stays that way, add an Assert.
Author: Laurenz Albe <laurenz.albe@cybertec.at>
Reported-By: Adam Warland <adam.warland@infor.com>
Reviewed-By: Heikki Linnakangas <hlinnaka@iki.fi>
Reviewed-By: Tom Lane <tgl@sss.pgh.pa.us>
Discussion: https://postgr.es/m/19341-1d9a22915edfec58%40postgresql.org
---
src/backend/utils/adt/varlena.c | 24 ++++++++++++-------
.../regress/expected/collate.icu.utf8.out | 7 ++++++
src/test/regress/sql/collate.icu.utf8.sql | 3 +++
3 files changed, 25 insertions(+), 9 deletions(-)
diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c
index 3894457ab40..26fbb9e969b 100644
--- a/src/backend/utils/adt/varlena.c
+++ b/src/backend/utils/adt/varlena.c
@@ -1110,10 +1110,10 @@ text_position_next_internal(char *start_ptr, TextPositionState *state)
const char *haystack_end = &haystack[haystack_len];
const char *hptr;
- Assert(start_ptr >= haystack && start_ptr <= haystack_end);
-
state->last_match_len_tmp = needle_len;
+ Assert(start_ptr >= haystack && start_ptr <= haystack_end && needle_len > 0);
+
if (!state->locale->deterministic)
{
/*
@@ -1123,9 +1123,9 @@ text_position_next_internal(char *start_ptr, TextPositionState *state)
* needle under the given collation.
*
* Note, the found substring could have a different length than the
- * needle, including being empty. Callers that want to skip over the
- * found string need to read the length of the found substring from
- * last_match_len rather than just using the length of their needle.
+ * needle. Callers that want to skip over the found string need to
+ * read the length of the found substring from last_match_len rather
+ * than just using the length of their needle.
*
* Most callers will require "greedy" semantics, meaning that we need
* to find the longest such substring, not the shortest. For callers
@@ -1136,6 +1136,8 @@ text_position_next_internal(char *start_ptr, TextPositionState *state)
hptr = start_ptr;
while (hptr < haystack_end)
{
+ const char *test_end;
+
/*
* First check the common case that there is a match in the
* haystack of exactly the length of the needle.
@@ -1146,11 +1148,14 @@ text_position_next_internal(char *start_ptr, TextPositionState *state)
return (char *) hptr;
/*
- * Else check if any of the possible substrings starting at hptr
- * are equal to the needle.
+ * Else check if any of the possible non-empty substrings starting
+ * at hptr are equal to the needle.
*/
- for (const char *test_end = hptr; test_end < haystack_end; test_end += pg_mblen(test_end))
+ test_end = hptr;
+ do
{
+ test_end += pg_mblen(test_end);
+
if (pg_strncoll(hptr, (test_end - hptr), needle, needle_len, state->locale) == 0)
{
state->last_match_len_tmp = (test_end - hptr);
@@ -1158,7 +1163,8 @@ text_position_next_internal(char *start_ptr, TextPositionState *state)
if (!state->greedy)
break;
}
- }
+ } while (test_end < haystack_end);
+
if (result_hptr)
break;
diff --git a/src/test/regress/expected/collate.icu.utf8.out b/src/test/regress/expected/collate.icu.utf8.out
index b8579a1efc6..0a14c1d93ff 100644
--- a/src/test/regress/expected/collate.icu.utf8.out
+++ b/src/test/regress/expected/collate.icu.utf8.out
@@ -1484,6 +1484,13 @@ SELECT array_sort('{a,B}'::text[] COLLATE "C");
{B,a}
(1 row)
+-- test replace() at the end of the string (bug #19341)
+SELECT replace('testX' COLLATE case_insensitive, 'x' COLLATE case_insensitive, 'es');
+ replace
+---------
+ testes
+(1 row)
+
-- test language tags
CREATE COLLATION lt_insensitive (provider = icu, locale = 'en-u-ks-level1', deterministic = false);
SELECT 'aBcD' COLLATE lt_insensitive = 'AbCd' COLLATE lt_insensitive;
diff --git a/src/test/regress/sql/collate.icu.utf8.sql b/src/test/regress/sql/collate.icu.utf8.sql
index 6f5abac0dc0..5e3fef9b381 100644
--- a/src/test/regress/sql/collate.icu.utf8.sql
+++ b/src/test/regress/sql/collate.icu.utf8.sql
@@ -568,6 +568,9 @@ SELECT 'abc' <= 'ABC' COLLATE case_insensitive, 'abc' >= 'ABC' COLLATE case_inse
SELECT array_sort('{a,B}'::text[] COLLATE case_insensitive);
SELECT array_sort('{a,B}'::text[] COLLATE "C");
+-- test replace() at the end of the string (bug #19341)
+SELECT replace('testX' COLLATE case_insensitive, 'x' COLLATE case_insensitive, 'es');
+
-- test language tags
CREATE COLLATION lt_insensitive (provider = icu, locale = 'en-u-ks-level1', deterministic = false);
SELECT 'aBcD' COLLATE lt_insensitive = 'AbCd' COLLATE lt_insensitive;
--
2.52.0
^ permalink raw reply [nested|flat] 16+ messages in thread
* Re: BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation
2025-12-02 10:03 BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation PG Bug reporting form <noreply@postgresql.org>
2025-12-02 16:24 ` Re: BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation Laurenz Albe <laurenz.albe@cybertec.at>
2025-12-02 16:36 ` Re: BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation Heikki Linnakangas <hlinnaka@iki.fi>
2025-12-02 17:18 ` Re: BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation Laurenz Albe <laurenz.albe@cybertec.at>
2025-12-02 17:25 ` Re: BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation Tom Lane <tgl@sss.pgh.pa.us>
2025-12-02 17:51 ` Re: BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation Laurenz Albe <laurenz.albe@cybertec.at>
2025-12-02 20:53 ` Re: BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation Tom Lane <tgl@sss.pgh.pa.us>
2025-12-03 07:51 ` Re: BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation Laurenz Albe <laurenz.albe@cybertec.at>
2025-12-03 15:12 ` Re: BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation Tom Lane <tgl@sss.pgh.pa.us>
2025-12-04 17:14 ` Re: BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation Laurenz Albe <laurenz.albe@cybertec.at>
@ 2025-12-06 01:12 ` Tom Lane <tgl@sss.pgh.pa.us>
2025-12-10 10:03 ` Re: BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation Laurenz Albe <laurenz.albe@cybertec.at>
0 siblings, 1 reply; 16+ messages in thread
From: Tom Lane @ 2025-12-06 01:12 UTC (permalink / raw)
To: Laurenz Albe <laurenz.albe@cybertec.at>; +Cc: Heikki Linnakangas <hlinnaka@iki.fi>; adam.warland@infor.com; pgsql-bugs@lists.postgresql.org
Laurenz Albe <laurenz.albe@cybertec.at> writes:
> On Wed, 2025-12-03 at 10:12 -0500, Tom Lane wrote:
>> We do require callers to eliminate the empty-needle case, and given
>> that I think we could assume that match substrings must be at least
>> 1 byte long. That assumption is what justifies the current API for
>> these functions, and perhaps we can also simplify this loop by
>> using it.
> The attached v5 patch simplifies the loop to a do-while loop, assuming
> that we cannot find a zero-length match.
> I have also updated the comments to no longer mention the possibility
> of an empty match, and for good measure I have added an Assert() that
> the needle cannot be empty.
LGTM. Pushed with tiny cosmetic fixes (mostly, more work on the
comments).
regards, tom lane
^ permalink raw reply [nested|flat] 16+ messages in thread
* Re: BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation
2025-12-02 10:03 BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation PG Bug reporting form <noreply@postgresql.org>
2025-12-02 16:24 ` Re: BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation Laurenz Albe <laurenz.albe@cybertec.at>
2025-12-02 16:36 ` Re: BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation Heikki Linnakangas <hlinnaka@iki.fi>
2025-12-02 17:18 ` Re: BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation Laurenz Albe <laurenz.albe@cybertec.at>
2025-12-02 17:25 ` Re: BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation Tom Lane <tgl@sss.pgh.pa.us>
2025-12-02 17:51 ` Re: BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation Laurenz Albe <laurenz.albe@cybertec.at>
2025-12-02 20:53 ` Re: BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation Tom Lane <tgl@sss.pgh.pa.us>
2025-12-03 07:51 ` Re: BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation Laurenz Albe <laurenz.albe@cybertec.at>
2025-12-03 15:12 ` Re: BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation Tom Lane <tgl@sss.pgh.pa.us>
2025-12-04 17:14 ` Re: BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation Laurenz Albe <laurenz.albe@cybertec.at>
2025-12-06 01:12 ` Re: BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation Tom Lane <tgl@sss.pgh.pa.us>
@ 2025-12-10 10:03 ` Laurenz Albe <laurenz.albe@cybertec.at>
0 siblings, 0 replies; 16+ messages in thread
From: Laurenz Albe @ 2025-12-10 10:03 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Heikki Linnakangas <hlinnaka@iki.fi>; adam.warland@infor.com; pgsql-bugs@lists.postgresql.org
On Fri, 2025-12-05 at 20:12 -0500, Tom Lane wrote:
> LGTM. Pushed with tiny cosmetic fixes (mostly, more work on the
> comments).
Thanks for all the tuition!
Yours,
Laurenz Albe
^ permalink raw reply [nested|flat] 16+ messages in thread
* Re: BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation
2025-12-02 10:03 BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation PG Bug reporting form <noreply@postgresql.org>
2025-12-02 16:24 ` Re: BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation Laurenz Albe <laurenz.albe@cybertec.at>
2025-12-02 16:36 ` Re: BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation Heikki Linnakangas <hlinnaka@iki.fi>
@ 2025-12-02 17:29 ` Heikki Linnakangas <hlinnaka@iki.fi>
2025-12-02 17:45 ` Re: BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation Laurenz Albe <laurenz.albe@cybertec.at>
1 sibling, 1 reply; 16+ messages in thread
From: Heikki Linnakangas @ 2025-12-02 17:29 UTC (permalink / raw)
To: Laurenz Albe <laurenz.albe@cybertec.at>; adam.warland@infor.com; pgsql-bugs@lists.postgresql.org
On 02/12/2025 18:36, Heikki Linnakangas wrote:
> On 02/12/2025 18:24, Laurenz Albe wrote:
>> On Tue, 2025-12-02 at 10:03 +0000, PG Bug reporting form wrote:
>>> PostgreSQL version: 18.1
>>>
>>> When using a nondeterministic ICU collation, the replace() function
>>> fails to
>>> replace a substring when that substring appears at the end of the input
>>> string.
>>>
>>> Occurrences of the same substring earlier in the string are replaced
>>> normally.
>>>
>>> Specific collation used:
>>> create collation test_nondeterministic (
>>> provider = icu,
>>> locale = 'und-u-ks-level2',
>>> deterministic = false
>>> )
>>>
>>> -- Replace final character under nondeterministic collation
>>> SELECT replace(
>>> 'testx' COLLATE "test_nondeterministic",
>>> 'x' COLLATE "test_nondeterministic",
>>> 'y') AS res1;
>>
>> I can reproduce the problem, and the attached patch fixes it for me.
>
> +1, looks good to me. Let's also add a regression test for this.
I added a simple test for this, and I think this is still not quite
right. I added the following to collate.icu.utf test:
CREATE TABLE test4nfd (a int, b text);
INSERT INTO test4nfd VALUES (1, 'cote'), (2, 'côte'), (3, 'coté'), (4,
'côté');
UPDATE test4nfd SET b = normalize(b, nfd);
-- This shows why replace should be greedy. Otherwise, in the NFD
-- case, the match would stop before the decomposed accents, which
-- would leave the accents in the results.
SELECT a, b, replace(b COLLATE ignore_accents, 'co', 'ma') FROM test4;
a | b | replace
---+------+---------
1 | cote | mate
2 | côte | mate
3 | coté | maté
4 | côté | maté
(4 rows)
SELECT a, b, replace(b COLLATE ignore_accents, 'co', 'ma') FROM test4nfd;
a | b | replace
---+------+---------
1 | cote | mate
2 | côte | mate
3 | coté | maté
4 | côté | maté
(4 rows)
+-- Test for match at the end of the string. (We had a bug on that
+-- once)
+SELECT a, b, replace(b COLLATE ignore_accents, 'te', 'ma') FROM test4nfd;
+ a | b | replace
+---+------+---------
+ 1 | cote | coma
+ 2 | côte | coma
+ 3 | coté | coma
+ 4 | côté | coma
+(4 rows)
+
In the added test query, the accents on the 'o' are stripped, which
doesn't look correct.
- Heikki
^ permalink raw reply [nested|flat] 16+ messages in thread
* Re: BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation
2025-12-02 10:03 BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation PG Bug reporting form <noreply@postgresql.org>
2025-12-02 16:24 ` Re: BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation Laurenz Albe <laurenz.albe@cybertec.at>
2025-12-02 16:36 ` Re: BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation Heikki Linnakangas <hlinnaka@iki.fi>
2025-12-02 17:29 ` Re: BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation Heikki Linnakangas <hlinnaka@iki.fi>
@ 2025-12-02 17:45 ` Laurenz Albe <laurenz.albe@cybertec.at>
0 siblings, 0 replies; 16+ messages in thread
From: Laurenz Albe @ 2025-12-02 17:45 UTC (permalink / raw)
To: Heikki Linnakangas <hlinnaka@iki.fi>; adam.warland@infor.com; pgsql-bugs@lists.postgresql.org
On Tue, 2025-12-02 at 19:29 +0200, Heikki Linnakangas wrote:
> I added a simple test for this, and I think this is still not quite
> right. I added the following to collate.icu.utf test:
>
> +-- Test for match at the end of the string. (We had a bug on that
> +-- once)
> +SELECT a, b, replace(b COLLATE ignore_accents, 'te', 'ma') FROM test4nfd;
> + a | b | replace
> +---+------+---------
> + 1 | cote | coma
> + 2 | côte | coma
> + 3 | coté | coma
> + 4 | côté | coma
> +(4 rows)
> +
>
> In the added test query, the accents on the 'o' are stripped, which
> doesn't look correct.
I am not sure if that is OK or not (after all, it's an accent
insensitive collation, so "coma" and "côma" should be the same).
But it seems unrelated to the bug report at hand.
Yours,
Laurenz Albe
^ permalink raw reply [nested|flat] 16+ messages in thread
end of thread, other threads:[~2025-12-10 10:03 UTC | newest]
Thread overview: 16+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2025-12-02 10:03 BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation PG Bug reporting form <noreply@postgresql.org>
2025-12-02 16:24 ` Laurenz Albe <laurenz.albe@cybertec.at>
2025-12-02 16:31 ` Laurenz Albe <laurenz.albe@cybertec.at>
2025-12-02 16:36 ` Heikki Linnakangas <hlinnaka@iki.fi>
2025-12-02 17:18 ` Laurenz Albe <laurenz.albe@cybertec.at>
2025-12-02 17:25 ` Tom Lane <tgl@sss.pgh.pa.us>
2025-12-02 17:51 ` Laurenz Albe <laurenz.albe@cybertec.at>
2025-12-02 20:53 ` Tom Lane <tgl@sss.pgh.pa.us>
2025-12-02 21:44 ` Laurenz Albe <laurenz.albe@cybertec.at>
2025-12-03 07:51 ` Laurenz Albe <laurenz.albe@cybertec.at>
2025-12-03 15:12 ` Tom Lane <tgl@sss.pgh.pa.us>
2025-12-04 17:14 ` Laurenz Albe <laurenz.albe@cybertec.at>
2025-12-06 01:12 ` Tom Lane <tgl@sss.pgh.pa.us>
2025-12-10 10:03 ` Laurenz Albe <laurenz.albe@cybertec.at>
2025-12-02 17:29 ` Heikki Linnakangas <hlinnaka@iki.fi>
2025-12-02 17:45 ` Laurenz Albe <laurenz.albe@cybertec.at>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox