pg.ddx.io  pgsql-committers@postgresql.org mailing list archive  
help / color / mirror / Atom feed
From: Jeff Davis <jdavis@postgresql.org>
To: pgsql-committers@lists.postgresql.org
Subject: pgsql: Canonicalize ICU locale names to language tags.
Date: Tue, 04 Apr 2023 17:45:28 +0000
Message-ID: <E1pjkil-001M8I-G2@gemulon.postgresql.org> (raw)

Canonicalize ICU locale names to language tags.

Convert to BCP47 language tags before storing in the catalog, except
during binary upgrade or when the locale comes from an existing
collation or template database.

The resulting language tags can vary slightly between ICU
versions. For instance, "@colBackwards=yes" is converted to
"und-u-kb-true" in older versions of ICU, and to the simpler (but
equivalent) "und-u-kb" in newer versions.

The process of canonicalizing to a language tag also understands more
input locale string formats than ucol_open(). For instance,
"fr_CA.UTF-8" is misinterpreted by ucol_open() and the region is
ignored; effectively treating it the same as the locale "fr" and
opening the wrong collator. Canonicalization properly interprets the
language and region, resulting in the language tag "fr-CA", which can
then be understood by ucol_open().

This commit fixes a problem in prior versions due to ucol_open()
misinterpreting locale strings as described above. For instance,
creating an ICU collation with locale "fr_CA.UTF-8" would store that
string directly in the catalog, which would later be passed to (and
misinterpreted by) ucol_open(). After this commit, the locale string
will be canonicalized to language tag "fr-CA" in the catalog, which
will be properly understood by ucol_open(). Because this fix affects
the resulting collator, we cannot change the locale string stored in
the catalog for existing databases or collations; otherwise we'd risk
corrupting indexes. Therefore, only canonicalize locales for
newly-created (not upgraded) collations/databases. For similar
reasons, do not backport.

Discussion: https://postgr.es/m/8c7af6820aed94dc7bc259d2aa7f9663518e6137.camel@j-davis.com
Reviewed-by: Peter Eisentraut

Branch
------
master

Details
-------
https://git.postgresql.org/pg/commitdiff/ea1db8ae70e5f4ceaae34dc9c06a07d59aaa022e

Modified Files
--------------
doc/src/sgml/charset.sgml                      |  2 +-
src/backend/commands/collationcmds.c           | 46 +++++++-------
src/backend/commands/dbcommands.c              | 20 ++++++
src/backend/utils/adt/pg_locale.c              | 85 ++++++++++++++++++++++++++
src/bin/initdb/initdb.c                        | 81 ++++++++++++++++++++++++
src/bin/initdb/t/001_initdb.pl                 |  2 +-
src/bin/pg_dump/t/002_pg_dump.pl               |  4 +-
src/include/utils/pg_locale.h                  |  1 +
src/test/regress/expected/collate.icu.utf8.out | 29 ++++++++-
src/test/regress/sql/collate.icu.utf8.sql      | 15 +++++
10 files changed, 258 insertions(+), 27 deletions(-)



Message-ID: <E1pjkil-001M8I-G2@gemulon.postgresql.org>
Permalink:  ../E1pjkil-001M8I-G2@gemulon.postgresql.org/
Also on:    postgresql.org/message-id/E1pjkil-001M8I-G2@gemulon.postgresql.org

 · 

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients using the --to and --cc options:
  reply via email

  To: pgsql-committers@postgresql.org
  Cc: jdavis@postgresql.org, pgsql-committers@lists.postgresql.org
  Subject: Re: pgsql: Canonicalize ICU locale names to language tags.
  In-Reply-To: <E1pjkil-001M8I-G2@gemulon.postgresql.org>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox