From: Marina Polyakova <m.polyakova@postgrespro.ru>
To: pgsql-hackers@postgresql.org
Cc: Justin Pryzby <pryzby@telsasoft.com>
Cc: Julien Rouhaud <rjuju123@gmail.com>
Cc: Daniel Verite <daniel@manitou-mail.org>
Cc: AndrewBille@gmail.com
Cc: michael@paquier.xyz
Cc: peter.eisentraut@enterprisedb.com
Subject: Re: ICU for global collation
Date: Wed, 14 Sep 2022 17:19:34 +0300
Message-ID: <6dd6db0984d86a51b7255ba79f111971@postgrespro.ru> (raw)
In-Reply-To: <38d2aac40a991b20adf032e509c6eff5@postgrespro.ru>
References: <20220628013205.GH19662@telsasoft.com>
<f385ba25e7f8be427b8c582e5cca7d79@postgrespro.ru>
<7d74b2d7-7fc6-f55b-3f77-c0e05569b64e@enterprisedb.com>
<YwRUn4J8+uqlyUYQ@paquier.xyz>
<7ad26354e75259f59c4a6c6997b8ee32@postgrespro.ru>
<YwWrVEDpLNGFA4Mq@paquier.xyz>
<20220824085923.jro5ao2m5b4mdll5@jrouhaud>
<13468bee-a5b4-65aa-932c-c12584f4458d@enterprisedb.com>
<Yw4KoBm1mS+2xzIj@paquier.xyz>
<7c99c132dc9c0ac630e0127f032ac480@postgrespro.ru>
<20220909164625.GN31833@telsasoft.com>
<60facfef2a8d54b227500260e86d823e@postgrespro.ru>
<1f86f120-7b02-68f6-a434-132d054afe33@enterprisedb.com>
<38d2aac40a991b20adf032e509c6eff5@postgrespro.ru>
Hello!
I was surprised that it is allowed to create clusters/databases where
the default ICU collations do not actually work due to unsupported
encodings:
$ initdb --encoding SQL_ASCII --locale-provider icu --icu-locale en-US
-D data &&
pg_ctl -D data -l logfile start &&
psql -c "SELECT 'a' < 'b'" template1
...
waiting for server to start.... done
server started
ERROR: encoding "SQL_ASCII" not supported by ICU
$ createdb --encoding SQL_ASCII --locale-provider icu --icu-locale en-US
--template template0 mydb &&
psql -c "SELECT 'a' < 'b'" mydb
ERROR: encoding "SQL_ASCII" not supported by ICU
The patch diff_check_icu_encoding.patch prohibits the creation of such
objects...
--
Marina Polyakova
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company
Attachments:
[text/x-diff] diff_check_icu_encoding.patch (3.2K, ../6dd6db0984d86a51b7255ba79f111971@postgrespro.ru/2-diff_check_icu_encoding.patch)
download | inline diff:diff --git a/src/backend/commands/dbcommands.c b/src/backend/commands/dbcommands.cindex 6ff48bb18f3639ae45d9528b32df51a4aebc60c0..07758d15e8613d5a049537ddf2c5992e57ad6424 100644--- a/src/backend/commands/dbcommands.c+++ b/src/backend/commands/dbcommands.c@@ -1042,11 +1042,16 @@ createdb(ParseState *pstate, const CreatedbStmt *stmt)
ereport(ERROR,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("ICU locale must be specified")));
- }- if (dblocprovider == COLLPROVIDER_ICU)
check_icu_locale(dbiculocale);
+ if (!(is_encoding_supported_by_icu(encoding)))+ ereport(ERROR,+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),+ errmsg("encoding \"%s\" is not supported with ICU provider",+ pg_encoding_to_char(encoding))));+ }+
/*
* Check that the new encoding and locale settings match the source
* database. We insist on this because we simply copy the source data ---
diff --git a/src/bin/initdb/initdb.c b/src/bin/initdb/initdb.cindex e00837ecacf4885cf2a176168c283f3e67c6eb53..8a762ced8340c9d8256f7832a4c19a43f1d5538a 100644--- a/src/bin/initdb/initdb.c+++ b/src/bin/initdb/initdb.c@@ -2362,6 +2362,16 @@ setup_locale_encoding(void)
if (!check_locale_encoding(lc_ctype, encodingid) ||
!check_locale_encoding(lc_collate, encodingid))
exit(1); /* check_locale_encoding printed the error */
++ if (locale_provider == COLLPROVIDER_ICU &&+ !(is_encoding_supported_by_icu(encodingid)))+ {+ pg_log_error("encoding \"%s\" is not supported with ICU provider",+ pg_encoding_to_char(encodingid));+ pg_log_error_hint("Rerun %s and choose a matching combination.",+ progname);+ exit(1);+ }
}
diff --git a/src/bin/initdb/t/001_initdb.pl b/src/bin/initdb/t/001_initdb.plindex a37f6dd9b334b6ee22d9fdd4d51422795cb54a39..e4bb3d0cdd9c23729c5fb97886374f8df558f239 100644--- a/src/bin/initdb/t/001_initdb.pl+++ b/src/bin/initdb/t/001_initdb.pl@@ -118,6 +118,15 @@ if ($ENV{with_icu} eq 'yes')
],
qr/FATAL: could not open collator for locale/,
'fails for invalid ICU locale');
++ command_fails_like(+ [+ 'initdb', '--no-sync',+ '--locale-provider=icu', '--icu-locale=en',+ '--encoding=SQL_ASCII', "$tempdir/dataX"+ ],+ qr/error: encoding "SQL_ASCII" is not supported with ICU provider/,+ 'encoding "SQL_ASCII" is not supported with ICU provider');
}
else
{
diff --git a/src/bin/scripts/t/020_createdb.pl b/src/bin/scripts/t/020_createdb.plindex e91c1d013d08d8bd1e3a92f2aba958c5c7713ca6..eaab3caa32669ead068719d98bb953c5c6ff5a17 100644--- a/src/bin/scripts/t/020_createdb.pl+++ b/src/bin/scripts/t/020_createdb.pl@@ -50,6 +50,16 @@ if ($ENV{with_icu} eq 'yes')
],
'fails for invalid ICU locale');
+ $node->command_fails_like(+ [+ 'createdb', '-T',+ 'template0', '--locale-provider=icu',+ '--icu-locale=en', '--encoding=SQL_ASCII',+ 'foobarX'+ ],+ qr/ERROR: encoding "SQL_ASCII" is not supported with ICU provider/,+ 'encoding "SQL_ASCII" is not supported with ICU provider');+
# additional node, which uses the icu provider
my $node2 = PostgreSQL::Test::Cluster->new('icu');
$node2->init(extra => ['--locale-provider=icu', '--icu-locale=en']);
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: m.polyakova@postgrespro.ru, pryzby@telsasoft.com, rjuju123@gmail.com, daniel@manitou-mail.org, AndrewBille@gmail.com, michael@paquier.xyz, peter.eisentraut@enterprisedb.com
Subject: Re: ICU for global collation
In-Reply-To: <6dd6db0984d86a51b7255ba79f111971@postgrespro.ru>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox