Subject: Supporting non-deterministic collations with tailoring rules.

From: Todd Lang <[email-masked]>
Date: Tue, 23 Sep 2025 14:51:28 +0000

Reposting this here from the Discord server as requested:

When creating a collation, in https://github.com/postgres/postgres/blob/master/src/backend/utils/adt/pg_locale_icu.c#L461 it is opening the collator with the tailoring rules supplied. However, it has hardcoded the strength level UCOL_DEFAULT_STRENGTH. This has the effect of ignoring the "deterministic=false" you may have specified in your CREATE COLLATION call. If, instead of UCOL_DEFAULT_STRENGTH, the code understood the deterministic parameter and passed either UCOL_PRIMARY for "deterministic=true", and UCOL_SECONDARY for "deterministic=false", this would preserve the attempt to obtain case-insensitivity in the locale while simultaneously allowing tailoring as expected.

I have made the modification to the pg_locale_icu.c and tested it locally (simply hardcoding UCOL_SECONDARY - not checking the deterministic parameter) and it behaves as expected, though I freely admit my knowledge of ICU intersecting with Postgres is rather limited.

---

From: "Daniel Verite" <[email-masked]>
Date: Wed, 24 Sep 2025 12:17:28 +0200

Todd Lang wrote:

> When creating a collation, in
> https://github.com/postgres/postgres/blob/master/src/backend/utils/adt/pg_locale_icu.c#L461
> it is opening the collator with the tailoring rules supplied. However, it
> has hardcoded the strength level UCOL_DEFAULT_STRENGTH. This has the effect
> of ignoring the "deterministic=false" you may have specified in your CREATE
> COLLATION call.

This is related to BUG #18771 previously reported at [1],
where the reporter notes that passing UCOL_DEFAULT works
for him whereas UCOL_DEFAULT_STRENGTH does not.
It looks like a documentation bug in ICU [2]
It says:

  strength: The default collation strength; one of UCOL_PRIMARY,
  UCOL_SECONDARY, UCOL_TERTIARY, UCOL_IDENTICAL,UCOL_DEFAULT_STRENGTH
  - can be also set in the rules.

But UCOL_DEFAULT_STRENGTH is an alias for UCOL_TERTIARY.
U_COL_DEFAULT is what should normally be passed to not override the
collation strength.

Now, by "it works", it means that the strength expressed in the rule
(with rules = '[strength 1]' in the case of the OP) takes effect.
This syntax is described at [3] (see "Rule Syntax" column)

There is a second problem: when the strength is specified in
the locale and not specified in the rules (as you did), it would also
be expected to take effect. It does not appear to be the case,
as if the rules were resetting the collation settings.
As mentioned in the thread at [1], Peter Eisentraut has submitted
this as a bug [4], but there hasn't been any follow-up to it
in 2.5 years.

> If, instead of UCOL_DEFAULT_STRENGTH, the code understood the
> deterministic parameter and passed either UCOL_PRIMARY for
> "deterministic=true", and UCOL_SECONDARY for "deterministic=false",
> this would preserve the attempt to obtain case-insensitivity in the
> locale while simultaneously allowing tailoring as expected.

We can't hardcode that deterministic=false implies that the strength
is 2. deterministic=false only says that the collation can have equal
strings that are not binary-equal.

To me, the most plausible fix on the Postgres side would be to pass
UCOL_DEFAULT instead of UCOL_DEFAULT_STRENGTH as in the attached,
which lets the user specify the strength in the rule, as the OP did in [1].

[1]:
https://www.postgresql.org/message-id/flat/18771-98bb23e455b0f367%40postgresql.org
[2]:
https://unicode-org.github.io/icu-docs/apidoc/released/icu4c/ucol_8h.html#a0cb1ddd81f322ed24e389f208eb35c8a
[3]: https://www.unicode.org/reports/tr35/tr35-collation.html#Setting_Options
[4]: https://unicode-org.atlassian.net/browse/ICU-22456

Best regards,
-- 
Daniel Vérité 
https://postgresql.verite.pro/
diff --git a/src/backend/utils/adt/pg_locale_icu.c b/src/backend/utils/adt/pg_locale_icu.c
index 96741e08269..e84ea5057e8 100644
--- a/src/backend/utils/adt/pg_locale_icu.c
+++ b/src/backend/utils/adt/pg_locale_icu.c
@@ -459,7 +459,7 @@ make_icu_collator(const char *iculocstr, const char *icurules)
 
 		status = U_ZERO_ERROR;
 		collator_all_rules = ucol_openRules(all_rules, u_strlen(all_rules),
-											UCOL_DEFAULT, UCOL_DEFAULT_STRENGTH,
+											UCOL_DEFAULT, UCOL_DEFAULT,
 											NULL, &status);
 		if (U_FAILURE(status))
 		{

---

From: Todd Lang <[email-masked]>
Date: Thu, 25 Sep 2025 13:32:17 +0000

Ah, somehow I missed your email on this.  This is, in fact, exactly what should happen.  The ICU folks are updating their documentation to reflect this with https://github.com/unicode-org/icu/pull/3684/files .

Is this small change a reasonable thing to include given the update in guidance from the ICU team?

-----Original Message-----
From: Daniel Verite <[email-masked]>
Sent: Wednesday, September 24, 2025 6:17 AM
To: Todd Lang <[email-masked]>
Cc: [email-masked]
Subject: Re: Supporting non-deterministic collations with tailoring rules.

CAUTION: This email originated from outside of D2L. Do not respond to, click links or open attachments unless you recognize the sender and know the content is safe.

        Todd Lang wrote:

> https://gith/
> ub.com%2Fpostgres%2Fpostgres%2Fblob%2Fmaster%2Fsrc%2Fbackend%2Futils%2
> Fadt%2Fpg_locale_icu.c%23L461&data=05%7C02%7CTodd.Lang%40D2L.com%7Cb34
> 6f047ed7944ebe01408ddfb5391b2%7C74bbca6d410b45b39b512a6aa6477079%7C0%7
> C0%7C638943058554088325%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydW
> UsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D
> %7C0%7C%7C%7C&sdata=4%2F6%2BalfTMFrnAzjQBt4i9Qa1BUUWBpaUJnGz%2B8dvy1s%
> 3D&reserved=0 it is opening the collator with the tailoring rules
> supplied. However, it has hardcoded the strength level
> UCOL_DEFAULT_STRENGTH. This has the effect of ignoring the
> "deterministic=false" you may have specified in your CREATE COLLATION
> call.

This is related to BUG #18771 previously reported at [1], where the reporter notes that passing UCOL_DEFAULT works for him whereas UCOL_DEFAULT_STRENGTH does not.
It looks like a documentation bug in ICU [2] It says:

But UCOL_DEFAULT_STRENGTH is an alias for UCOL_TERTIARY.
U_COL_DEFAULT is what should normally be passed to not override the collation strength.

Now, by "it works", it means that the strength expressed in the rule (with rules = '[strength 1]' in the case of the OP) takes effect.
This syntax is described at [3] (see "Rule Syntax" column)

There is a second problem: when the strength is specified in the locale and not specified in the rules (as you did), it would also be expected to take effect. It does not appear to be the case, as if the rules were resetting the collation settings.
As mentioned in the thread at [1], Peter Eisentraut has submitted this as a bug [4], but there hasn't been any follow-up to it in 2.5 years.

> locale while simultaneously allowing tailoring as expected.

We can't hardcode that deterministic=false implies that the strength is 2. deterministic=false only says that the collation can have equal strings that are not binary-equal.

To me, the most plausible fix on the Postgres side would be to pass UCOL_DEFAULT instead of UCOL_DEFAULT_STRENGTH as in the attached, which lets the user specify the strength in the rule, as the OP did in [1].

[1]:

Best regards,

---

From: "Daniel Verite" <[email-masked]>
Date: Fri, 26 Sep 2025 16:28:26 +0200

Todd Lang wrote:

> The ICU folks are updating their documentation to reflect
> this with https://github.com/unicode-org/icu/pull/3684/files .
> 

I think so. Added to the commitfest:

https://commitfest.postgresql.org/patch/6084/

Best regards,

---

From: Peter Eisentraut <[email-masked]>
Date: Thu, 12 Mar 2026 11:00:32 +0100

> which lets the user specify the strength in the rule, as the OP did in [1].

With this change, I don't see that the bug reported in ICU-22456 is 
fixed.  See attached my test case.

What change of behavior are you expecting from your patch?  Should there 
be test cases?
From 2588c3eb6ff4acdf2aaeba0ebe8b026b811d7755 Mon Sep 17 00:00:00 2001
From: Peter Eisentraut <[email-masked]>
Date: Thu, 12 Mar 2026 10:58:12 +0100
Subject: [PATCH] Test for: collation customization with rules loses attributes
 of original collation

---
 src/backend/utils/adt/pg_locale_icu.c     | 2 +-
 src/test/regress/sql/collate.icu.utf8.sql | 6 ++++++
 2 files changed, 7 insertions(+), 1 deletion(-)

diff --git a/src/backend/utils/adt/pg_locale_icu.c b/src/backend/utils/adt/pg_locale_icu.c
index 352b4c3885f..5ad05fcd016 100644
--- a/src/backend/utils/adt/pg_locale_icu.c
+++ b/src/backend/utils/adt/pg_locale_icu.c
@@ -587,7 +587,7 @@ make_icu_collator(const char *iculocstr, const char *icurules)
 
 		status = U_ZERO_ERROR;
 		collator_all_rules = ucol_openRules(all_rules, u_strlen(all_rules),
-											UCOL_DEFAULT, UCOL_DEFAULT_STRENGTH,
+											UCOL_DEFAULT, UCOL_DEFAULT,
 											NULL, &status);
 		if (U_FAILURE(status))
 		{
diff --git a/src/test/regress/sql/collate.icu.utf8.sql b/src/test/regress/sql/collate.icu.utf8.sql
index b6c54503d21..929e9603089 100644
--- a/src/test/regress/sql/collate.icu.utf8.sql
+++ b/src/test/regress/sql/collate.icu.utf8.sql
@@ -513,6 +513,12 @@ CREATE TABLE test7 (a text);
 
 CREATE COLLATION testcoll_rulesx (provider = icu, locale = '', rules = '!!wrong!!');
 
+-- WIP https://unicode-org.atlassian.net/browse/ICU-22456
+CREATE COLLATION testcoll_rules2 (provider = icu, locale = 'und-u-ka-shifted-ks-level1', deterministic = false, rules = '');
+CREATE COLLATION testcoll_rules2nr (provider = icu, locale = 'und-u-ka-shifted-ks-level1', deterministic = false);  -- the same without rules
+SELECT 'ab' = 'a b' COLLATE testcoll_rules2nr;  -- true
+SELECT 'ab' = 'a b' COLLATE testcoll_rules2;  -- FIXME should be true
+
 
 -- nondeterministic collations
 
-- 
2.53.0

---

From: "Daniel Verite" <[email-masked]>
Date: Sun, 15 Mar 2026 19:52:24 +0100

[resent for the list]

> What change of behavior are you expecting from your patch?  Should there 

Indeed it does not fix the behavior reported in ICU-22456
(=collation properties being "reset" when adding rules)
It fixes the fact that specificying the strength in the rule itself
will be taken into account instead of the strength being
forced to tertiary.

PFA a new patch with a specific test case added.

With regards to reports on pgsql-bugs, it fixes #18771
and the second part of #19425.
It does not fix #19045, which looks the same as ICU-22456
It also does not fix the first part of #19425 (which looks the
same as #19045). We cannot fix that in Postgres AFAIU.
However adding the strength or other collation properties in
the rules can be used as a workaround against the ICU-22456
issue, provided the attached change is made.

Best regards,
 
 		status = U_ZERO_ERROR;
 		collator_all_rules = ucol_openRules(all_rules, u_strlen(all_rules),
-											UCOL_DEFAULT, UCOL_DEFAULT_STRENGTH,
+											UCOL_DEFAULT, UCOL_DEFAULT,
 											NULL, &status);
 		if (U_FAILURE(status))
 		{
diff --git a/src/test/regress/expected/collate.icu.utf8.out b/src/test/regress/expected/collate.icu.utf8.out
index 1325e123877..e4eff391946 100644
--- a/src/test/regress/expected/collate.icu.utf8.out
+++ b/src/test/regress/expected/collate.icu.utf8.out
@@ -1297,6 +1297,16 @@ DROP TABLE test7;
 CREATE COLLATION testcoll_rulesx (provider = icu, locale = '', rules = '!!wrong!!');
 NOTICE:  using standard form "und" for ICU locale ""
 ERROR:  could not open collator for locale "und" with rules "!!wrong!!": U_INVALID_FORMAT_ERROR
+-- strength specified in the rules
+CREATE COLLATION strength_in_rule ( provider = icu, locale='und',
+ deterministic=false, rules='[strength 1]');
+SELECT 'a'='à' COLLATE strength_in_rule; -- true because of the rule
+ ?column? 
+----------
+ t
+(1 row)
+
+--
 -- nondeterministic collations
 CREATE COLLATION ctest_det (provider = icu, locale = '', deterministic = true);
 NOTICE:  using standard form "und" for ICU locale ""
diff --git a/src/test/regress/sql/collate.icu.utf8.sql b/src/test/regress/sql/collate.icu.utf8.sql
index b6c54503d21..c0613b238b8 100644
--- a/src/test/regress/sql/collate.icu.utf8.sql
+++ b/src/test/regress/sql/collate.icu.utf8.sql
@@ -513,6 +513,12 @@ DROP TABLE test7;

+-- strength specified in the rules
+CREATE COLLATION strength_in_rule ( provider = icu, locale='und',
+ deterministic=false, rules='[strength 1]');
+SELECT 'a'='à' COLLATE strength_in_rule; -- true because of the rule
+
+--

---

From: Peter Eisentraut <[email-masked]>
Date: Wed, 18 Mar 2026 09:12:33 +0100

> issue, provided the attached change is made.

Ok, committed.

Thoughts on backpatching?  Seems like a straightforward bug fix.

---

From: "Daniel Verite" <[email-masked]>
Date: Wed, 18 Mar 2026 19:21:52 +0100

> Thoughts on backpatching?  Seems like a straightforward bug fix.

Thanks for pushing the patch!
I thought it was not a candidate for backpatching because it's
a collation change.
Now, to have a problem with that change in a minor release,
a user would need to have an ICU collation with a rule containing
"[strength N]", where N<>3, even though it was ineffective (so
either the user did not notice or they did but kept it
nonetheless). Users who reported the problem hoped for a backpatch,
but we cannot say there is zero risk with that change in a minor
release. Personally I'm +0.

Best regards,

---

From: Peter Eisentraut <[email-masked]>
Date: Thu, 19 Mar 2026 06:57:29 +0100

> release. Personally I'm +0.

Ok, let's not then.