public inbox for [email protected]
help / color / mirror / Atom feedFrom: Jeff Davis <[email protected]>
To: Thomas Munro <[email protected]>
To: Joe Conway <[email protected]>
Cc: Peter Eisentraut <[email protected]>
Cc: Jeremy Schneider <[email protected]>
Cc: Peter Geoghegan <[email protected]>
Cc: Nasby, Jim <[email protected]>
Cc: Tom Lane <[email protected]>
Cc: pgsql-hackers <[email protected]>
Subject: Re: Collation version tracking for macOS
Date: Wed, 07 Dec 2022 21:56:38 -0800
Message-ID: <[email protected]> (raw)
In-Reply-To: <CA+hUKGLEqMhnpZrgAcisoUeYFGz8W6EWdhtK2h-4QN0iOSFRqw@mail.gmail.com>
References: <[email protected]>
<[email protected]>
<[email protected]>
<CA+hUKGL36vXMfcaDq+U1ZkoSsdfFnNx7GxhGM7aYzEbKs1W0=Q@mail.gmail.com>
<CA+hUKGKq=iLH3bY+nK7v8b2zBCuKOk-fe0cP0it2RxNaWFVxYA@mail.gmail.com>
<[email protected]>
<CA+hUKG+OSQtrRAk-bHwMJmuvqp-b-LGuxsF2PoDBVyQcT+VEAQ@mail.gmail.com>
<[email protected]>
<CA+hUKGJxg6AbKC9RJ7r1ByVLtvVkThQV+RZO6BKVWYESPCp3Ug@mail.gmail.com>
<[email protected]>
<CA+hUKGJhPeW3iRqO3XQB=+BTMx7O6zTCFKTfeSV9TXBtdyMqbQ@mail.gmail.com>
<CA+hUKGLr=9d+-k8PVv8e__TOxuq=n0SKNDpqCzbGrK5EbDyxAg@mail.gmail.com>
<[email protected]>
<CA+hUKGKWX6q_UXsDhVbhYns2+VsrPLW5o6ohqO6C=cZ6r23G7w@mail.gmail.com>
<[email protected]>
<[email protected]>
<CA+hUKGLEqMhnpZrgAcisoUeYFGz8W6EWdhtK2h-4QN0iOSFRqw@mail.gmail.com>
On Tue, 2022-12-06 at 10:33 +1300, Thomas Munro wrote:
> OK, I'm going to see what happens if I try to wrangle that stuff into
> a new catalogue table.
I've been hacking on a major refactor of the locale-related code. I
attached my progress and I think several patches are ready.
The main motivation is that I was frustrated by the special cases
everywhere. I wanted to make it easier to hack on this code going
forward, now that we are adding even more complexity for multiple ICU
libraries.
I'm posting to this thread (rather than my previous refactoring
thread[1]) because a lot of the people interested in working on this
code are here. So, if you like (or don't like) the structure of these
changes, please let me know.
Changes:
* Introduce pg_locale_internal.h to hide all USE_ICU code,
including all callers of the ICU routines. The files that
still need to include pg_locale_internal.h are:
- pg_locale.c
- regc_pg_locale.c
- formatting.c
- like.c
- like_support.c
- collationcmds.c
* Other callers (in files that don't include
pg_locale_internal.h) don't need to branch based on the
provider, platform, database encoding, USE_ICU,
HAVE_LOCALE_T, etc.
* ICU and libc are treated the same way in more places.
* I made it so pg_locale_t is constructed first, then
moved to TopMemoryContext, so that it won't leak in
TopMemoryContext if errors are encountered.
* Introduce pg_strcoll, pg_strncoll, pg_strxfrm, and pg_strnxfrm
so that varlena/hash/verchar code doesn't worry about the
details.
* Add method structure pg_icu_library, borrowed from Thomas's
patch, that provides one convenient place to provide
multiple-ICU-library support.
* Add a hook that allows you to fill in the pg_icu_library
structure however you want while a pg_locale_t is being
constructed. This allows do-it-yourself ICU library
lockdown.
On the negative side, it increases the line count. Part of that is
because adding indirection for the ICU library is just more lines of
code, but a lot of it is just that I used a lot of smaller functions.
Perhaps my style is a bit verbose?
Even though we're close to consensus on how we should offer control
over the ICU libraries, having the hook may be useful for
experimentation, testing, or as a last resort. Right now the hook has
limited information to use to find the right library -- just the ICU
collation name and the version, because that's what we have in the
catalog. But I assume the patch Thomas is working on will change that.
Performance:
I did brief performance sanity tests on several paths and the results
are unremarkable (which is generally good for a refactor). On the path
I was watching most closely, ICU/UTF8/en-US-x-icu, it came in about 2%
faster, which was a pleasant surprise. This was true both when I
disabled abbreviated keys (to stress localized comparison paths) and
also with abbreviated keys enabled. My previous refactoring work[1]
ended up a percent or two slower. My guess right now is that I moved
some code around after I noticed that ICU accepts NUL-terminated
strings (by specifying the lenght as -1), and that helped. But I'll
need to profile and look more closely to be more certain of my results,
these are preliminary.
There are a few things that could be done differently:
* I am still a bit confused about why someone would want a collation
with a different lc_collate and lc_ctype in libc; and assuming there is
a reason, why it can't be done with ICU. The way I did the refactoring
tries to accommodate them as different concepts, but I can rip that
out.
* In theory, we could also support multilib libc, and an associated
get_libc_library() and hook, but there are a couple big challenges.
Firstly, if it's the default locale, it relies on setlocale(), so we'd
have to figure out what to do about that. Second, having an a second
version of glibc on your system is not as normal or trivial as having a
second version of ICU.
* I made the hook simple, but all it can do is replace the ICU
library. It's possible that it would want to construct it's own entire
pg_locale_t for some reason, and keep more complex state in a private
pointer, or something like that. It seemed better to keep it simple,
but maybe someone would want more flexibility there?
* I used the library indirection for pretty much all ICU calls,
including the ucnv_ and the uloc_ functions. I did this mainly because,
if we are so paranoid about ICU changing in subtle ways, we might as
well make it possible to lock down everything. I can rip this out, too,
but it didn't add many lines.
Loose ends:
* I need to do something with get_collation_actual_version. I had an
earlier iteration that went through pg_newlocale() and then queried the
resulting pg_locale_t structure, but that changed the error paths in a
way that failed a couple tests, so I left that out.
* Error paths could be improved further to make sure that libc
locale_t and UCollator structures are freed in error paths during
construction. I was thinking about using resowner for this, and then if
the pg_locale_t structure gets moved to TopMemoryContext, just doing a
ResourceOwnerForget. Alternatively, I could just be careful about the
error paths.
* We'd need to adapt this and make sure it works with whatever scheme
we decide is best for finding the right library. I suspect this would
just be adding another parameter to get_icu_library (and the hook) to
represent the new collation provider Oid (and get_icu_library could use
that to look up the library names and load them).
Comments welcome.
[1]
https://www.postgresql.org/message-id/[email protected]
--
Jeff Davis
PostgreSQL Contributor Team - AWS
Attachments:
[text/x-patch] v2-0001-Add-pg_strcoll-and-pg_strncoll.patch (19.3K, ../[email protected]/2-v2-0001-Add-pg_strcoll-and-pg_strncoll.patch)
download | inline diff:
From 5d144e19686a0ae01e1d80a9f8c356079f41b9eb Mon Sep 17 00:00:00 2001
From: Jeff Davis <[email protected]>
Date: Thu, 1 Dec 2022 14:45:15 -0800
Subject: [PATCH v2 1/6] Add pg_strcoll() and pg_strncoll().
Callers with NUL-terminated strings should call the former; callers
with strings and their length should call the latter.
---
src/backend/utils/adt/pg_locale.c | 420 ++++++++++++++++++++++++++++--
src/backend/utils/adt/varlena.c | 230 +---------------
src/include/utils/pg_locale.h | 3 +
3 files changed, 406 insertions(+), 247 deletions(-)
diff --git a/src/backend/utils/adt/pg_locale.c b/src/backend/utils/adt/pg_locale.c
index 2b42d9ccd8..6cd629ecb4 100644
--- a/src/backend/utils/adt/pg_locale.c
+++ b/src/backend/utils/adt/pg_locale.c
@@ -79,6 +79,12 @@
#include <shlwapi.h>
#endif
+/*
+ * This should be large enough that most strings will fit, but small enough
+ * that we feel comfortable putting it on the stack
+ */
+#define TEXTBUFLEN 1024
+
#define MAX_L10N_DATA 80
@@ -123,6 +129,19 @@ static char *IsoLocaleName(const char *);
#endif
#ifdef USE_ICU
+/*
+ * Converter object for converting between ICU's UChar strings and C strings
+ * in database encoding. Since the database encoding doesn't change, we only
+ * need one of these per session.
+ */
+static UConverter *icu_converter = NULL;
+
+static void init_icu_converter(void);
+static size_t uchar_length(UConverter *converter,
+ const char *str, size_t len);
+static int32_t uchar_convert(UConverter *converter,
+ UChar *dest, int32_t destlen,
+ const char *str, size_t srclen);
static void icu_set_collation_attributes(UCollator *collator, const char *loc);
#endif
@@ -1731,15 +1750,356 @@ get_collation_actual_version(char collprovider, const char *collcollate)
return collversion;
}
+/*
+ * pg_strncoll_libc_win32_utf8
+ *
+ * Win32 does not have UTF-8. Convert UTF8 arguments to wide characters and
+ * invoke wcscoll() or wcscoll_l().
+ */
+#ifdef WIN32
+static int
+pg_strncoll_libc_win32_utf8(const char *arg1, size_t len1, const char *arg2,
+ size_t len2, pg_locale_t locale)
+{
+ char sbuf[TEXTBUFLEN];
+ char *buf = sbuf;
+ char *a1p,
+ *a2p;
+ int a1len = len1 * 2 + 2;
+ int a2len = len2 * 2 + 2;
+ int r;
+ int result;
+
+ Assert(!locale || locale->provider == COLLPROVIDER_LIBC);
+ Assert(GetDatabaseEncoding() == PG_UTF8);
+#ifndef WIN32
+ Assert(false);
+#endif
+
+ if (a1len + a2len > TEXTBUFLEN)
+ buf = palloc(a1len + a2len);
+
+ a1p = buf;
+ a2p = buf + a1len;
+
+ /* API does not work for zero-length input */
+ if (len1 == 0)
+ r = 0;
+ else
+ {
+ r = MultiByteToWideChar(CP_UTF8, 0, arg1, len1,
+ (LPWSTR) a1p, a1len / 2);
+ if (!r)
+ ereport(ERROR,
+ (errmsg("could not convert string to UTF-16: error code %lu",
+ GetLastError())));
+ }
+ ((LPWSTR) a1p)[r] = 0;
+
+ if (len2 == 0)
+ r = 0;
+ else
+ {
+ r = MultiByteToWideChar(CP_UTF8, 0, arg2, len2,
+ (LPWSTR) a2p, a2len / 2);
+ if (!r)
+ ereport(ERROR,
+ (errmsg("could not convert string to UTF-16: error code %lu",
+ GetLastError())));
+ }
+ ((LPWSTR) a2p)[r] = 0;
+
+ errno = 0;
+#ifdef HAVE_LOCALE_T
+ if (locale)
+ result = wcscoll_l((LPWSTR) a1p, (LPWSTR) a2p, locale->info.lt);
+ else
+#endif
+ result = wcscoll((LPWSTR) a1p, (LPWSTR) a2p);
+ if (result == 2147483647) /* _NLSCMPERROR; missing from mingw
+ * headers */
+ ereport(ERROR,
+ (errmsg("could not compare Unicode strings: %m")));
+
+ if (buf != sbuf)
+ pfree(buf);
+
+ return result;
+}
+#endif /* WIN32 */
+
+/*
+ * pg_strcoll_libc
+ *
+ * Call strcoll(), strcoll_l(), wcscoll(), or wcscoll_l() as appropriate for
+ * the given locale, platform, and database encoding. If the locale is NULL,
+ * use the database collation.
+ *
+ * Arguments must be encoded in the database encoding and nul-terminated.
+ */
+static int
+pg_strcoll_libc(const char *arg1, const char *arg2, pg_locale_t locale)
+{
+ int result;
+
+ Assert(!locale || locale->provider == COLLPROVIDER_LIBC);
+#ifdef WIN32
+ if (GetDatabaseEncoding() == PG_UTF8)
+ {
+ size_t len1 = strlen(arg1);
+ size_t len2 = strlen(arg2);
+ result = pg_strncoll_libc_win32_utf8(arg1, len1, arg2, len2, locale);
+ }
+ else
+#endif /* WIN32 */
+ if (locale)
+ {
+#ifdef HAVE_LOCALE_T
+ result = strcoll_l(arg1, arg2, locale->info.lt);
+#else
+ /* shouldn't happen */
+ elog(ERROR, "unsupported collprovider: %c", locale->provider);
+#endif
+ }
+ else
+ result = strcoll(arg1, arg2);
+
+ return result;
+}
+
+/*
+ * pg_strncoll_libc
+ *
+ * Null-terminate the arguments and call pg_strcoll_libc().
+ */
+static int
+pg_strncoll_libc(const char *arg1, size_t len1, const char *arg2, size_t len2,
+ pg_locale_t locale)
+{
+ char sbuf[TEXTBUFLEN];
+ char *buf = sbuf;
+ size_t bufsize1 = len1 + 1;
+ size_t bufsize2 = len2 + 1;
+ char *arg1n;
+ char *arg2n;
+ int result;
+
+ Assert(!locale || locale->provider == COLLPROVIDER_LIBC);
+
+#ifdef WIN32
+ /* check for this case before doing the work for nul-termination */
+ if (GetDatabaseEncoding() == PG_UTF8)
+ return pg_strncoll_libc_win32_utf8(arg1, len1, arg2, len2, locale);
+#endif /* WIN32 */
+
+ if (bufsize1 + bufsize2 > TEXTBUFLEN)
+ buf = palloc(bufsize1 + bufsize2);
+
+ arg1n = buf;
+ arg2n = buf + bufsize1;
+
+ /* nul-terminate arguments */
+ memcpy(arg1n, arg1, len1);
+ arg1n[len1] = '\0';
+ memcpy(arg2n, arg2, len2);
+ arg2n[len2] = '\0';
+
+ result = pg_strcoll_libc(arg1n, arg2n, locale);
+
+ if (buf != sbuf)
+ pfree(buf);
+
+ return result;
+}
#ifdef USE_ICU
+
/*
- * Converter object for converting between ICU's UChar strings and C strings
- * in database encoding. Since the database encoding doesn't change, we only
- * need one of these per session.
+ * pg_strncoll_icu_no_utf8
+ *
+ * Convert the arguments from the database encoding to UChar strings, then
+ * call ucol_strcoll().
+ *
+ * When the database encoding is UTF-8, and ICU supports ucol_strcollUTF8(),
+ * caller should call that instead.
*/
-static UConverter *icu_converter = NULL;
+static int
+pg_strncoll_icu_no_utf8(const char *arg1, size_t len1,
+ const char *arg2, size_t len2, pg_locale_t locale)
+{
+ char sbuf[TEXTBUFLEN];
+ char *buf = sbuf;
+ int32_t ulen1;
+ int32_t ulen2;
+ size_t bufsize1;
+ size_t bufsize2;
+ UChar *uchar1,
+ *uchar2;
+ int result;
+
+ Assert(locale->provider == COLLPROVIDER_ICU);
+#ifdef HAVE_UCOL_STRCOLLUTF8
+ Assert(GetDatabaseEncoding() != PG_UTF8);
+#endif
+
+ init_icu_converter();
+
+ ulen1 = uchar_length(icu_converter, arg1, len1);
+ ulen2 = uchar_length(icu_converter, arg2, len2);
+
+ bufsize1 = (ulen1 + 1) * sizeof(UChar);
+ bufsize2 = (ulen2 + 1) * sizeof(UChar);
+
+ if (bufsize1 + bufsize2 > TEXTBUFLEN)
+ buf = palloc(bufsize1 + bufsize2);
+
+ uchar1 = (UChar *) buf;
+ uchar2 = (UChar *) (buf + bufsize1);
+ ulen1 = uchar_convert(icu_converter, uchar1, ulen1 + 1, arg1, len1);
+ ulen2 = uchar_convert(icu_converter, uchar2, ulen2 + 1, arg2, len2);
+
+ result = ucol_strcoll(locale->info.icu.ucol,
+ uchar1, ulen1,
+ uchar2, ulen2);
+
+ if (buf != sbuf)
+ pfree(buf);
+
+ return result;
+}
+
+/*
+ * pg_strncoll_icu
+ *
+ * Call ucol_strcollUTF8() or ucol_strcoll() as appropriate for the given
+ * database encoding.
+ *
+ * Arguments must be encoded in the database encoding.
+ */
+static int
+pg_strncoll_icu(const char *arg1, size_t len1, const char *arg2, size_t len2,
+ pg_locale_t locale)
+{
+ int result;
+
+ Assert(locale->provider == COLLPROVIDER_ICU);
+
+#ifdef HAVE_UCOL_STRCOLLUTF8
+ if (GetDatabaseEncoding() == PG_UTF8)
+ {
+ UErrorCode status;
+
+ status = U_ZERO_ERROR;
+ result = ucol_strcollUTF8(locale->info.icu.ucol,
+ arg1, len1,
+ arg2, len2,
+ &status);
+ if (U_FAILURE(status))
+ ereport(ERROR,
+ (errmsg("collation failed: %s", u_errorName(status))));
+ }
+ else
+#endif
+ {
+ result = pg_strncoll_icu_no_utf8(arg1, len1, arg2, len2, locale);
+ }
+
+ return result;
+}
+
+/*
+ * pg_strcoll_icu
+ *
+ * Calculate the string lengths and call pg_strncoll_icu().
+ */
+static int
+pg_strcoll_icu(const char *arg1, const char *arg2, pg_locale_t locale)
+{
+ Assert(locale->provider == COLLPROVIDER_ICU);
+ return pg_strncoll_icu(arg1, -1, arg2, -1, locale);
+}
+
+#endif /* USE_ICU */
+
+/*
+ * pg_strcoll
+ *
+ * Call ucol_strcollUTF8(), ucol_strcoll(), strcoll(), strcoll_l(), wcscoll(),
+ * or wcscoll_l() as appropriate for the given locale, platform, and database
+ * encoding. If the locale is not specified, use the database collation.
+ *
+ * Arguments must be encoded in the database encoding and nul-terminated.
+ *
+ * If the collation is deterministic, break ties with strcmp().
+ */
+int
+pg_strcoll(const char *arg1, const char *arg2, pg_locale_t locale)
+{
+ int result;
+
+ if (!locale || locale->provider == COLLPROVIDER_LIBC)
+ result = pg_strcoll_libc(arg1, arg2, locale);
+#ifdef USE_ICU
+ else if (locale->provider == COLLPROVIDER_ICU)
+ result = pg_strcoll_icu(arg1, arg2, locale);
+#endif
+ else
+ /* shouldn't happen */
+ elog(ERROR, "unsupported collprovider: %c", locale->provider);
+
+ /* Break tie if necessary. */
+ if (result == 0 && (!locale || locale->deterministic))
+ result = strcmp(arg1, arg2);
+
+ return result;
+}
+
+/*
+ * pg_strncoll
+ *
+ * Call ucol_strcollUTF8(), ucol_strcoll(), strcoll(), strcoll_l(), wcscoll(),
+ * or wcscoll_l() as appropriate for the given locale, platform, and database
+ * encoding. If the locale is not specified, use the database collation.
+ *
+ * Arguments must be encoded in the database encoding.
+ *
+ * If the collation is deterministic, break ties with memcmp(), and then with
+ * the string length.
+ *
+ * This function may need to nul-terminate the arguments for libc functions;
+ * so if the caller already has nul-terminated strings, it should call
+ * pg_strcoll() instead.
+ */
+int
+pg_strncoll(const char *arg1, size_t len1, const char *arg2, size_t len2,
+ pg_locale_t locale)
+{
+ int result;
+
+ if (!locale || locale->provider == COLLPROVIDER_LIBC)
+ result = pg_strncoll_libc(arg1, len1, arg2, len2, locale);
+#ifdef USE_ICU
+ else if (locale->provider == COLLPROVIDER_ICU)
+ result = pg_strncoll_icu(arg1, len1, arg2, len2, locale);
+#endif
+ else
+ /* shouldn't happen */
+ elog(ERROR, "unsupported collprovider: %c", locale->provider);
+
+ /* Break tie if necessary. */
+ if (result == 0 && (!locale || locale->deterministic))
+ {
+ result = memcmp(arg1, arg2, Min(len1, len2));
+ if ((result == 0) && (len1 != len2))
+ result = (len1 < len2) ? -1 : 1;
+ }
+
+ return result;
+}
+
+
+#ifdef USE_ICU
static void
init_icu_converter(void)
{
@@ -1767,6 +2127,39 @@ init_icu_converter(void)
icu_converter = conv;
}
+/*
+ * Find length, in UChars, of given string if converted to UChar string.
+ */
+static size_t
+uchar_length(UConverter *converter, const char *str, size_t len)
+{
+ UErrorCode status = U_ZERO_ERROR;
+ int32_t ulen;
+ ulen = ucnv_toUChars(converter, NULL, 0, str, len, &status);
+ if (U_FAILURE(status) && status != U_BUFFER_OVERFLOW_ERROR)
+ ereport(ERROR,
+ (errmsg("%s failed: %s", "ucnv_toUChars", u_errorName(status))));
+ return ulen;
+}
+
+/*
+ * Convert the given source string into a UChar string, stored in dest, and
+ * return the length (in UChars).
+ */
+static int32_t
+uchar_convert(UConverter *converter, UChar *dest, int32_t destlen,
+ const char *src, size_t srclen)
+{
+ UErrorCode status = U_ZERO_ERROR;
+ int32_t ulen;
+ status = U_ZERO_ERROR;
+ ulen = ucnv_toUChars(converter, dest, destlen, src, srclen, &status);
+ if (U_FAILURE(status))
+ ereport(ERROR,
+ (errmsg("%s failed: %s", "ucnv_toUChars", u_errorName(status))));
+ return ulen;
+}
+
/*
* Convert a string in the database encoding into a string of UChars.
*
@@ -1782,26 +2175,15 @@ init_icu_converter(void)
int32_t
icu_to_uchar(UChar **buff_uchar, const char *buff, size_t nbytes)
{
- UErrorCode status;
- int32_t len_uchar;
+ int32_t len_uchar;
init_icu_converter();
- status = U_ZERO_ERROR;
- len_uchar = ucnv_toUChars(icu_converter, NULL, 0,
- buff, nbytes, &status);
- if (U_FAILURE(status) && status != U_BUFFER_OVERFLOW_ERROR)
- ereport(ERROR,
- (errmsg("%s failed: %s", "ucnv_toUChars", u_errorName(status))));
+ len_uchar = uchar_length(icu_converter, buff, nbytes);
*buff_uchar = palloc((len_uchar + 1) * sizeof(**buff_uchar));
-
- status = U_ZERO_ERROR;
- len_uchar = ucnv_toUChars(icu_converter, *buff_uchar, len_uchar + 1,
- buff, nbytes, &status);
- if (U_FAILURE(status))
- ereport(ERROR,
- (errmsg("%s failed: %s", "ucnv_toUChars", u_errorName(status))));
+ len_uchar = uchar_convert(icu_converter,
+ *buff_uchar, len_uchar + 1, buff, nbytes);
return len_uchar;
}
diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c
index c5e7ee7ca2..c904bc0825 100644
--- a/src/backend/utils/adt/varlena.c
+++ b/src/backend/utils/adt/varlena.c
@@ -1535,10 +1535,6 @@ varstr_cmp(const char *arg1, int len1, const char *arg2, int len2, Oid collid)
}
else
{
- char a1buf[TEXTBUFLEN];
- char a2buf[TEXTBUFLEN];
- char *a1p,
- *a2p;
pg_locale_t mylocale;
mylocale = pg_newlocale_from_collation(collid);
@@ -1555,171 +1551,7 @@ varstr_cmp(const char *arg1, int len1, const char *arg2, int len2, Oid collid)
if (len1 == len2 && memcmp(arg1, arg2, len1) == 0)
return 0;
-#ifdef WIN32
- /* Win32 does not have UTF-8, so we need to map to UTF-16 */
- if (GetDatabaseEncoding() == PG_UTF8
- && (!mylocale || mylocale->provider == COLLPROVIDER_LIBC))
- {
- int a1len;
- int a2len;
- int r;
-
- if (len1 >= TEXTBUFLEN / 2)
- {
- a1len = len1 * 2 + 2;
- a1p = palloc(a1len);
- }
- else
- {
- a1len = TEXTBUFLEN;
- a1p = a1buf;
- }
- if (len2 >= TEXTBUFLEN / 2)
- {
- a2len = len2 * 2 + 2;
- a2p = palloc(a2len);
- }
- else
- {
- a2len = TEXTBUFLEN;
- a2p = a2buf;
- }
-
- /* stupid Microsloth API does not work for zero-length input */
- if (len1 == 0)
- r = 0;
- else
- {
- r = MultiByteToWideChar(CP_UTF8, 0, arg1, len1,
- (LPWSTR) a1p, a1len / 2);
- if (!r)
- ereport(ERROR,
- (errmsg("could not convert string to UTF-16: error code %lu",
- GetLastError())));
- }
- ((LPWSTR) a1p)[r] = 0;
-
- if (len2 == 0)
- r = 0;
- else
- {
- r = MultiByteToWideChar(CP_UTF8, 0, arg2, len2,
- (LPWSTR) a2p, a2len / 2);
- if (!r)
- ereport(ERROR,
- (errmsg("could not convert string to UTF-16: error code %lu",
- GetLastError())));
- }
- ((LPWSTR) a2p)[r] = 0;
-
- errno = 0;
-#ifdef HAVE_LOCALE_T
- if (mylocale)
- result = wcscoll_l((LPWSTR) a1p, (LPWSTR) a2p, mylocale->info.lt);
- else
-#endif
- result = wcscoll((LPWSTR) a1p, (LPWSTR) a2p);
- if (result == 2147483647) /* _NLSCMPERROR; missing from mingw
- * headers */
- ereport(ERROR,
- (errmsg("could not compare Unicode strings: %m")));
-
- /* Break tie if necessary. */
- if (result == 0 &&
- (!mylocale || mylocale->deterministic))
- {
- result = memcmp(arg1, arg2, Min(len1, len2));
- if ((result == 0) && (len1 != len2))
- result = (len1 < len2) ? -1 : 1;
- }
-
- if (a1p != a1buf)
- pfree(a1p);
- if (a2p != a2buf)
- pfree(a2p);
-
- return result;
- }
-#endif /* WIN32 */
-
- if (len1 >= TEXTBUFLEN)
- a1p = (char *) palloc(len1 + 1);
- else
- a1p = a1buf;
- if (len2 >= TEXTBUFLEN)
- a2p = (char *) palloc(len2 + 1);
- else
- a2p = a2buf;
-
- memcpy(a1p, arg1, len1);
- a1p[len1] = '\0';
- memcpy(a2p, arg2, len2);
- a2p[len2] = '\0';
-
- if (mylocale)
- {
- if (mylocale->provider == COLLPROVIDER_ICU)
- {
-#ifdef USE_ICU
-#ifdef HAVE_UCOL_STRCOLLUTF8
- if (GetDatabaseEncoding() == PG_UTF8)
- {
- UErrorCode status;
-
- status = U_ZERO_ERROR;
- result = ucol_strcollUTF8(mylocale->info.icu.ucol,
- arg1, len1,
- arg2, len2,
- &status);
- if (U_FAILURE(status))
- ereport(ERROR,
- (errmsg("collation failed: %s", u_errorName(status))));
- }
- else
-#endif
- {
- int32_t ulen1,
- ulen2;
- UChar *uchar1,
- *uchar2;
-
- ulen1 = icu_to_uchar(&uchar1, arg1, len1);
- ulen2 = icu_to_uchar(&uchar2, arg2, len2);
-
- result = ucol_strcoll(mylocale->info.icu.ucol,
- uchar1, ulen1,
- uchar2, ulen2);
-
- pfree(uchar1);
- pfree(uchar2);
- }
-#else /* not USE_ICU */
- /* shouldn't happen */
- elog(ERROR, "unsupported collprovider: %c", mylocale->provider);
-#endif /* not USE_ICU */
- }
- else
- {
-#ifdef HAVE_LOCALE_T
- result = strcoll_l(a1p, a2p, mylocale->info.lt);
-#else
- /* shouldn't happen */
- elog(ERROR, "unsupported collprovider: %c", mylocale->provider);
-#endif
- }
- }
- else
- result = strcoll(a1p, a2p);
-
- /* Break tie if necessary. */
- if (result == 0 &&
- (!mylocale || mylocale->deterministic))
- result = strcmp(a1p, a2p);
-
- if (a1p != a1buf)
- pfree(a1p);
- if (a2p != a2buf)
- pfree(a2p);
+ result = pg_strncoll(arg1, len1, arg2, len2, mylocale);
}
return result;
@@ -2377,65 +2209,7 @@ varstrfastcmp_locale(char *a1p, int len1, char *a2p, int len2, SortSupport ssup)
return sss->last_returned;
}
- if (sss->locale)
- {
- if (sss->locale->provider == COLLPROVIDER_ICU)
- {
-#ifdef USE_ICU
-#ifdef HAVE_UCOL_STRCOLLUTF8
- if (GetDatabaseEncoding() == PG_UTF8)
- {
- UErrorCode status;
-
- status = U_ZERO_ERROR;
- result = ucol_strcollUTF8(sss->locale->info.icu.ucol,
- a1p, len1,
- a2p, len2,
- &status);
- if (U_FAILURE(status))
- ereport(ERROR,
- (errmsg("collation failed: %s", u_errorName(status))));
- }
- else
-#endif
- {
- int32_t ulen1,
- ulen2;
- UChar *uchar1,
- *uchar2;
-
- ulen1 = icu_to_uchar(&uchar1, a1p, len1);
- ulen2 = icu_to_uchar(&uchar2, a2p, len2);
-
- result = ucol_strcoll(sss->locale->info.icu.ucol,
- uchar1, ulen1,
- uchar2, ulen2);
-
- pfree(uchar1);
- pfree(uchar2);
- }
-#else /* not USE_ICU */
- /* shouldn't happen */
- elog(ERROR, "unsupported collprovider: %c", sss->locale->provider);
-#endif /* not USE_ICU */
- }
- else
- {
-#ifdef HAVE_LOCALE_T
- result = strcoll_l(sss->buf1, sss->buf2, sss->locale->info.lt);
-#else
- /* shouldn't happen */
- elog(ERROR, "unsupported collprovider: %c", sss->locale->provider);
-#endif
- }
- }
- else
- result = strcoll(sss->buf1, sss->buf2);
-
- /* Break tie if necessary. */
- if (result == 0 &&
- (!sss->locale || sss->locale->deterministic))
- result = strcmp(sss->buf1, sss->buf2);
+ result = pg_strcoll(sss->buf1, sss->buf2, sss->locale);
/* Cache result, perhaps saving an expensive strcoll() call next time */
sss->cache_blob = false;
diff --git a/src/include/utils/pg_locale.h b/src/include/utils/pg_locale.h
index a875942123..bf70ae08ca 100644
--- a/src/include/utils/pg_locale.h
+++ b/src/include/utils/pg_locale.h
@@ -100,6 +100,9 @@ extern void make_icu_collator(const char *iculocstr,
extern pg_locale_t pg_newlocale_from_collation(Oid collid);
extern char *get_collation_actual_version(char collprovider, const char *collcollate);
+extern int pg_strcoll(const char *arg1, const char *arg2, pg_locale_t locale);
+extern int pg_strncoll(const char *arg1, size_t len1,
+ const char *arg2, size_t len2, pg_locale_t locale);
#ifdef USE_ICU
extern int32_t icu_to_uchar(UChar **buff_uchar, const char *buff, size_t nbytes);
--
2.34.1
[text/x-patch] v2-0002-Add-pg_strxfrm-and-pg_strxfrm_prefix.patch (24.2K, ../[email protected]/3-v2-0002-Add-pg_strxfrm-and-pg_strxfrm_prefix.patch)
download | inline diff:
From bbee706cfa8a074c31429b178242c77084af9d6c Mon Sep 17 00:00:00 2001
From: Jeff Davis <[email protected]>
Date: Thu, 1 Dec 2022 14:41:38 -0800
Subject: [PATCH v2 2/6] Add pg_strxfrm() and pg_strxfrm_prefix().
Callers with a NUL-terminated string should call the former; callers
with a string and length should call the latter.
---
src/backend/access/hash/hashfunc.c | 45 ++--
src/backend/utils/adt/pg_locale.c | 382 +++++++++++++++++++++++++++++
src/backend/utils/adt/varchar.c | 41 ++--
src/backend/utils/adt/varlena.c | 142 +++--------
src/include/utils/pg_locale.h | 10 +
5 files changed, 470 insertions(+), 150 deletions(-)
diff --git a/src/backend/access/hash/hashfunc.c b/src/backend/access/hash/hashfunc.c
index f890f79ee1..b8136e496f 100644
--- a/src/backend/access/hash/hashfunc.c
+++ b/src/backend/access/hash/hashfunc.c
@@ -291,21 +291,19 @@ hashtext(PG_FUNCTION_ARGS)
#ifdef USE_ICU
if (mylocale->provider == COLLPROVIDER_ICU)
{
- int32_t ulen = -1;
- UChar *uchar = NULL;
- Size bsize;
- uint8_t *buf;
+ Size bsize, rsize;
+ char *buf;
+ const char *keydata = VARDATA_ANY(key);
+ size_t keylen = VARSIZE_ANY_EXHDR(key);
- ulen = icu_to_uchar(&uchar, VARDATA_ANY(key), VARSIZE_ANY_EXHDR(key));
-
- bsize = ucol_getSortKey(mylocale->info.icu.ucol,
- uchar, ulen, NULL, 0);
+ bsize = pg_strnxfrm(NULL, 0, keydata, keylen, mylocale);
buf = palloc(bsize);
- ucol_getSortKey(mylocale->info.icu.ucol,
- uchar, ulen, buf, bsize);
- pfree(uchar);
- result = hash_any(buf, bsize);
+ rsize = pg_strnxfrm(buf, bsize, keydata, keylen, mylocale);
+ if (rsize != bsize)
+ elog(ERROR, "pg_strnxfrm() returned unexpected result");
+
+ result = hash_any((uint8_t *) buf, bsize);
pfree(buf);
}
@@ -349,21 +347,20 @@ hashtextextended(PG_FUNCTION_ARGS)
#ifdef USE_ICU
if (mylocale->provider == COLLPROVIDER_ICU)
{
- int32_t ulen = -1;
- UChar *uchar = NULL;
- Size bsize;
- uint8_t *buf;
+ Size bsize, rsize;
+ char *buf;
+ const char *keydata = VARDATA_ANY(key);
+ size_t keylen = VARSIZE_ANY_EXHDR(key);
- ulen = icu_to_uchar(&uchar, VARDATA_ANY(key), VARSIZE_ANY_EXHDR(key));
-
- bsize = ucol_getSortKey(mylocale->info.icu.ucol,
- uchar, ulen, NULL, 0);
+ bsize = pg_strnxfrm(NULL, 0, keydata, keylen, mylocale);
buf = palloc(bsize);
- ucol_getSortKey(mylocale->info.icu.ucol,
- uchar, ulen, buf, bsize);
- pfree(uchar);
- result = hash_any_extended(buf, bsize, PG_GETARG_INT64(1));
+ rsize = pg_strnxfrm(buf, bsize, keydata, keylen, mylocale);
+ if (rsize != bsize)
+ elog(ERROR, "pg_strnxfrm() returned unexpected result");
+
+ result = hash_any_extended((uint8_t *) buf, bsize,
+ PG_GETARG_INT64(1));
pfree(buf);
}
diff --git a/src/backend/utils/adt/pg_locale.c b/src/backend/utils/adt/pg_locale.c
index 6cd629ecb4..133bb03a13 100644
--- a/src/backend/utils/adt/pg_locale.c
+++ b/src/backend/utils/adt/pg_locale.c
@@ -2099,6 +2099,388 @@ pg_strncoll(const char *arg1, size_t len1, const char *arg2, size_t len2,
}
+static size_t
+pg_strxfrm_libc(char *dest, const char *src, size_t destsize,
+ pg_locale_t locale)
+{
+ Assert(!locale || locale->provider == COLLPROVIDER_LIBC);
+
+#ifdef TRUST_STXFRM
+#ifdef HAVE_LOCALE_T
+ if (locale)
+ return strxfrm_l(dest, src, destsize, locale->info.lt);
+ else
+#endif
+ return strxfrm(dest, src, destsize);
+#else
+ /* shouldn't happen */
+ elog(ERROR, "unsupported collprovider: %c", locale->provider);
+#endif
+}
+
+static size_t
+pg_strnxfrm_libc(char *dest, const char *src, size_t srclen, size_t destsize,
+ pg_locale_t locale)
+{
+ char sbuf[TEXTBUFLEN];
+ char *buf = sbuf;
+ size_t bufsize = srclen + 1;
+ size_t result;
+
+ Assert(!locale || locale->provider == COLLPROVIDER_LIBC);
+
+ if (bufsize > TEXTBUFLEN)
+ buf = palloc(bufsize);
+
+ /* nul-terminate arguments */
+ memcpy(buf, src, srclen);
+ buf[srclen] = '\0';
+
+ result = pg_strxfrm_libc(dest, buf, destsize, locale);
+
+ if (buf != sbuf)
+ pfree(buf);
+
+ return result;
+}
+
+static size_t
+pg_strxfrm_prefix_libc(char *dest, const char *src, size_t destsize,
+ pg_locale_t locale)
+{
+ Assert(!locale || locale->provider == COLLPROVIDER_LIBC);
+ /* unsupported; shouldn't happen */
+ elog(ERROR, "collprovider '%c' does not support pg_strxfrm_prefix()",
+ locale->provider);
+}
+
+static size_t
+pg_strnxfrm_prefix_libc(char *dest, const char *src, size_t srclen,
+ size_t destsize, pg_locale_t locale)
+{
+ Assert(!locale || locale->provider == COLLPROVIDER_LIBC);
+ /* unsupported; shouldn't happen */
+ elog(ERROR, "collprovider '%c' does not support pg_strnxfrm_prefix()",
+ locale->provider);
+}
+
+#ifdef USE_ICU
+
+static size_t
+pg_strnxfrm_icu(char *dest, const char *src, size_t srclen, size_t destsize,
+ pg_locale_t locale)
+{
+ char sbuf[TEXTBUFLEN];
+ char *buf = sbuf;
+ UChar *uchar;
+ int32_t ulen;
+ size_t uchar_bsize;
+ Size result_bsize;
+
+ Assert(locale->provider == COLLPROVIDER_ICU);
+
+ init_icu_converter();
+
+ ulen = uchar_length(icu_converter, src, srclen);
+
+ uchar_bsize = (ulen + 1) * sizeof(UChar);
+
+ if (uchar_bsize > TEXTBUFLEN)
+ buf = palloc(uchar_bsize);
+
+ uchar = (UChar *) buf;
+
+ ulen = uchar_convert(icu_converter, uchar, ulen + 1, src, srclen);
+
+ result_bsize = ucol_getSortKey(locale->info.icu.ucol,
+ uchar, ulen,
+ (uint8_t *) dest, destsize);
+
+ if (buf != sbuf)
+ pfree(buf);
+
+ return result_bsize;
+}
+
+static size_t
+pg_strxfrm_icu(char *dest, const char *src, size_t destsize,
+ pg_locale_t locale)
+{
+ Assert(locale->provider == COLLPROVIDER_ICU);
+ return pg_strnxfrm_icu(dest, src, -1, destsize, locale);
+}
+
+static size_t
+pg_strnxfrm_prefix_icu_no_utf8(char *dest, const char *src, size_t srclen,
+ size_t destsize, pg_locale_t locale)
+{
+ char sbuf[TEXTBUFLEN];
+ char *buf = sbuf;
+ UCharIterator iter;
+ uint32_t state[2];
+ UErrorCode status;
+ int32_t ulen = -1;
+ UChar *uchar = NULL;
+ size_t uchar_bsize;
+ Size result_bsize;
+
+ Assert(locale->provider == COLLPROVIDER_ICU);
+ Assert(GetDatabaseEncoding() != PG_UTF8);
+
+ init_icu_converter();
+
+ ulen = uchar_length(icu_converter, src, srclen);
+
+ uchar_bsize = (ulen + 1) * sizeof(UChar);
+
+ if (uchar_bsize > TEXTBUFLEN)
+ buf = palloc(uchar_bsize);
+
+ uchar = (UChar *) buf;
+
+ ulen = uchar_convert(icu_converter, uchar, ulen + 1, src, srclen);
+
+ uiter_setString(&iter, uchar, ulen);
+ state[0] = state[1] = 0; /* won't need that again */
+ status = U_ZERO_ERROR;
+ result_bsize = ucol_nextSortKeyPart(locale->info.icu.ucol,
+ &iter,
+ state,
+ (uint8_t *) dest,
+ destsize,
+ &status);
+ if (U_FAILURE(status))
+ ereport(ERROR,
+ (errmsg("sort key generation failed: %s",
+ u_errorName(status))));
+
+ return result_bsize;
+}
+
+static size_t
+pg_strnxfrm_prefix_icu(char *dest, const char *src, size_t srclen,
+ size_t destsize, pg_locale_t locale)
+{
+ size_t result;
+
+ Assert(locale->provider == COLLPROVIDER_ICU);
+
+ if (GetDatabaseEncoding() == PG_UTF8)
+ {
+ UCharIterator iter;
+ uint32_t state[2];
+ UErrorCode status;
+
+ uiter_setUTF8(&iter, src, srclen);
+ state[0] = state[1] = 0; /* won't need that again */
+ status = U_ZERO_ERROR;
+ result = ucol_nextSortKeyPart(locale->info.icu.ucol,
+ &iter,
+ state,
+ (uint8_t *) dest,
+ destsize,
+ &status);
+ if (U_FAILURE(status))
+ ereport(ERROR,
+ (errmsg("sort key generation failed: %s",
+ u_errorName(status))));
+ }
+ else
+ result = pg_strnxfrm_prefix_icu_no_utf8(dest, src, srclen, destsize,
+ locale);
+
+ return result;
+}
+
+static size_t
+pg_strxfrm_prefix_icu(char *dest, const char *src, size_t destsize,
+ pg_locale_t locale)
+{
+ Assert(locale->provider == COLLPROVIDER_ICU);
+ return pg_strnxfrm_prefix_icu(dest, src, -1, destsize, locale);
+}
+
+#endif
+
+/*
+ * Return true if the collation provider supports pg_strxfrm() and
+ * pg_strnxfrm(); otherwise false.
+ *
+ * Unfortunately, it seems that strxfrm() for non-C collations is broken on
+ * many common platforms; testing of multiple versions of glibc reveals that,
+ * for many locales, strcoll() and strxfrm() do not return consistent
+ * results. While no other libc other than Cygwin has so far been shown to
+ * have a problem, we take the conservative course of action for right now and
+ * disable this categorically. (Users who are certain this isn't a problem on
+ * their system can define TRUST_STRXFRM.)
+ */
+bool
+pg_strxfrm_enabled(pg_locale_t locale)
+{
+ if (!locale || locale->provider == COLLPROVIDER_LIBC)
+ {
+#ifdef TRUST_STRXFRM
+ return true;
+#else
+ return false;
+#endif
+ }
+ else if (locale->provider == COLLPROVIDER_ICU)
+ return true;
+ else
+ /* shouldn't happen */
+ elog(ERROR, "unsupported collprovider: %c", locale->provider);
+}
+
+/*
+ * pg_strxfrm
+ *
+ * Transforms 'src' to a nul-terminated string stored in 'dest' such that
+ * ordinary strcmp() on transformed strings is equivalent to pg_strcoll() on
+ * untransformed strings.
+ *
+ * The provided 'src' must be nul-terminated.
+ *
+ * If destsize is large enough to hold the result, returns the number of bytes
+ * copied to 'dest'; otherwise, returns the number of bytes needed to hold the
+ * result and leaves the contents of 'dest' undefined. If destsize is zero,
+ * 'dest' may be NULL.
+ */
+size_t
+pg_strxfrm(char *dest, const char *src, size_t destsize, pg_locale_t locale)
+{
+ size_t result;
+
+ if (!locale || locale->provider == COLLPROVIDER_LIBC)
+ result = pg_strxfrm_libc(dest, src, destsize, locale);
+#ifdef USE_ICU
+ else if (locale->provider == COLLPROVIDER_ICU)
+ result = pg_strxfrm_icu(dest, src, destsize, locale);
+#endif
+ else
+ /* shouldn't happen */
+ elog(ERROR, "unsupported collprovider: %c", locale->provider);
+
+ return result;
+}
+
+/*
+ * pg_strnxfrm
+ *
+ * Transforms 'src' to a nul-terminated string stored in 'dest' such that
+ * ordinary strcmp() on transformed strings is equivalent to pg_strcoll() on
+ * untransformed strings.
+ *
+ * If destsize is large enough to hold the result, returns the number of bytes
+ * copied to 'dest'; otherwise, returns the number of bytes needed to hold the
+ * result and leaves the contents of 'dest' undefined. If destsize is zero,
+ * 'dest' may be NULL.
+ *
+ * This function may need to nul-terminate the argument for libc functions;
+ * so if the caller already has a nul-terminated string, it should call
+ * pg_strxfrm() instead.
+ */
+size_t
+pg_strnxfrm(char *dest, size_t destsize, const char *src, size_t srclen,
+ pg_locale_t locale)
+{
+ size_t result;
+
+ if (!locale || locale->provider == COLLPROVIDER_LIBC)
+ result = pg_strnxfrm_libc(dest, src, srclen, destsize, locale);
+#ifdef USE_ICU
+ else if (locale->provider == COLLPROVIDER_ICU)
+ result = pg_strnxfrm_icu(dest, src, srclen, destsize, locale);
+#endif
+ else
+ /* shouldn't happen */
+ elog(ERROR, "unsupported collprovider: %c", locale->provider);
+
+ return result;
+}
+
+/*
+ * Return true if the collation provider supports pg_strxfrm_prefix() and
+ * pg_strnxfrm_prefix(); otherwise false.
+ */
+bool
+pg_strxfrm_prefix_enabled(pg_locale_t locale)
+{
+ if (!locale || locale->provider == COLLPROVIDER_LIBC)
+ return false;
+ else if (locale->provider == COLLPROVIDER_ICU)
+ return true;
+ else
+ /* shouldn't happen */
+ elog(ERROR, "unsupported collprovider: %c", locale->provider);
+}
+
+/*
+ * pg_strxfrm_prefix
+ *
+ * Transforms 'src' to a byte sequence stored in 'dest' such that ordinary
+ * memcmp() on the byte sequence is equivalent to pg_strcoll() on
+ * untransformed strings. The result is not nul-terminated.
+ *
+ * The provided 'src' must be nul-terminated.
+ *
+ * If destsize is not large enough to hold the entire result, stores just the
+ * prefix in 'dest'. Returns the number of bytes actually copied to 'dest'.
+ */
+size_t
+pg_strxfrm_prefix(char *dest, const char *src, size_t destsize,
+ pg_locale_t locale)
+{
+ size_t result;
+
+ if (!locale || locale->provider == COLLPROVIDER_LIBC)
+ result = pg_strxfrm_prefix_libc(dest, src, destsize, locale);
+#ifdef USE_ICU
+ else if (locale->provider == COLLPROVIDER_ICU)
+ result = pg_strxfrm_prefix_icu(dest, src, destsize, locale);
+#endif
+ else
+ /* shouldn't happen */
+ elog(ERROR, "unsupported collprovider: %c", locale->provider);
+
+ return result;
+}
+
+/*
+ * pg_strnxfrm_prefix
+ *
+ * Transforms 'src' to a byte sequence stored in 'dest' such that ordinary
+ * memcmp() on the byte sequence is equivalent to pg_strcoll() on
+ * untransformed strings. The result is not nul-terminated.
+ *
+ * The provided 'src' must be nul-terminated.
+ *
+ * If destsize is not large enough to hold the entire result, stores just the
+ * prefix in 'dest'. Returns the number of bytes actually copied to 'dest'.
+ *
+ * This function may need to nul-terminate the argument for libc functions;
+ * so if the caller already has a nul-terminated string, it should call
+ * pg_strxfrm_prefix() instead.
+ */
+size_t
+pg_strnxfrm_prefix(char *dest, size_t destsize, const char *src,
+ size_t srclen, pg_locale_t locale)
+{
+ size_t result;
+
+ if (!locale || locale->provider == COLLPROVIDER_LIBC)
+ result = pg_strnxfrm_prefix_libc(dest, src, srclen, destsize, locale);
+#ifdef USE_ICU
+ else if (locale->provider == COLLPROVIDER_ICU)
+ result = pg_strnxfrm_prefix_icu(dest, src, srclen, destsize, locale);
+#endif
+ else
+ /* shouldn't happen */
+ elog(ERROR, "unsupported collprovider: %c", locale->provider);
+
+ return result;
+}
+
#ifdef USE_ICU
static void
init_icu_converter(void)
diff --git a/src/backend/utils/adt/varchar.c b/src/backend/utils/adt/varchar.c
index a63c498181..d0bc528e9f 100644
--- a/src/backend/utils/adt/varchar.c
+++ b/src/backend/utils/adt/varchar.c
@@ -1019,21 +1019,17 @@ hashbpchar(PG_FUNCTION_ARGS)
#ifdef USE_ICU
if (mylocale->provider == COLLPROVIDER_ICU)
{
- int32_t ulen = -1;
- UChar *uchar = NULL;
- Size bsize;
- uint8_t *buf;
+ Size bsize, rsize;
+ char *buf;
- ulen = icu_to_uchar(&uchar, keydata, keylen);
-
- bsize = ucol_getSortKey(mylocale->info.icu.ucol,
- uchar, ulen, NULL, 0);
+ bsize = pg_strnxfrm(NULL, 0, keydata, keylen, mylocale);
buf = palloc(bsize);
- ucol_getSortKey(mylocale->info.icu.ucol,
- uchar, ulen, buf, bsize);
- pfree(uchar);
- result = hash_any(buf, bsize);
+ rsize = pg_strnxfrm(buf, bsize, keydata, keylen, mylocale);
+ if (rsize != bsize)
+ elog(ERROR, "pg_strnxfrm() returned unexpected result");
+
+ result = hash_any((uint8_t *) buf, bsize);
pfree(buf);
}
@@ -1081,21 +1077,18 @@ hashbpcharextended(PG_FUNCTION_ARGS)
#ifdef USE_ICU
if (mylocale->provider == COLLPROVIDER_ICU)
{
- int32_t ulen = -1;
- UChar *uchar = NULL;
- Size bsize;
- uint8_t *buf;
+ Size bsize, rsize;
+ char *buf;
- ulen = icu_to_uchar(&uchar, keydata, keylen);
-
- bsize = ucol_getSortKey(mylocale->info.icu.ucol,
- uchar, ulen, NULL, 0);
+ bsize = pg_strnxfrm(NULL, 0, keydata, keylen, mylocale);
buf = palloc(bsize);
- ucol_getSortKey(mylocale->info.icu.ucol,
- uchar, ulen, buf, bsize);
- pfree(uchar);
- result = hash_any_extended(buf, bsize, PG_GETARG_INT64(1));
+ rsize = pg_strnxfrm(buf, bsize, keydata, keylen, mylocale);
+ if (rsize != bsize)
+ elog(ERROR, "pg_strnxfrm() returned unexpected result");
+
+ result = hash_any_extended((uint8_t *) buf, bsize,
+ PG_GETARG_INT64(1));
pfree(buf);
}
diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c
index c904bc0825..2dfba4b488 100644
--- a/src/backend/utils/adt/varlena.c
+++ b/src/backend/utils/adt/varlena.c
@@ -1887,20 +1887,6 @@ varstr_sortsupport(SortSupport ssup, Oid typid, Oid collid)
*/
locale = pg_newlocale_from_collation(collid);
- /*
- * There is a further exception on Windows. When the database
- * encoding is UTF-8 and we are not using the C collation, complex
- * hacks are required. We don't currently have a comparator that
- * handles that case, so we fall back on the slow method of having the
- * sort code invoke bttextcmp() (in the case of text) via the fmgr
- * trampoline. ICU locales work just the same on Windows, however.
- */
-#ifdef WIN32
- if (GetDatabaseEncoding() == PG_UTF8 &&
- !(locale && locale->provider == COLLPROVIDER_ICU))
- return;
-#endif
-
/*
* We use varlenafastcmp_locale except for type NAME.
*/
@@ -1916,13 +1902,7 @@ varstr_sortsupport(SortSupport ssup, Oid typid, Oid collid)
/*
* Unfortunately, it seems that abbreviation for non-C collations is
- * broken on many common platforms; testing of multiple versions of glibc
- * reveals that, for many locales, strcoll() and strxfrm() do not return
- * consistent results, which is fatal to this optimization. While no
- * other libc other than Cygwin has so far been shown to have a problem,
- * we take the conservative course of action for right now and disable
- * this categorically. (Users who are certain this isn't a problem on
- * their system can define TRUST_STRXFRM.)
+ * broken on many common platforms; see pg_strxfrm_enabled().
*
* Even apart from the risk of broken locales, it's possible that there
* are platforms where the use of abbreviated keys should be disabled at
@@ -1935,10 +1915,8 @@ varstr_sortsupport(SortSupport ssup, Oid typid, Oid collid)
* categorically, we may still want or need to disable it for particular
* platforms.
*/
-#ifndef TRUST_STRXFRM
- if (!collate_c && !(locale && locale->provider == COLLPROVIDER_ICU))
+ if (!collate_c && !pg_strxfrm_enabled(locale))
abbreviate = false;
-#endif
/*
* If we're using abbreviated keys, or if we're using a locale-aware
@@ -2227,6 +2205,7 @@ varstrfastcmp_locale(char *a1p, int len1, char *a2p, int len2, SortSupport ssup)
static Datum
varstr_abbrev_convert(Datum original, SortSupport ssup)
{
+ const size_t max_prefix_bytes = sizeof(Datum);
VarStringSortSupport *sss = (VarStringSortSupport *) ssup->ssup_extra;
VarString *authoritative = DatumGetVarStringPP(original);
char *authoritative_data = VARDATA_ANY(authoritative);
@@ -2239,7 +2218,7 @@ varstr_abbrev_convert(Datum original, SortSupport ssup)
pres = (char *) &res;
/* memset(), so any non-overwritten bytes are NUL */
- memset(pres, 0, sizeof(Datum));
+ memset(pres, 0, max_prefix_bytes);
len = VARSIZE_ANY_EXHDR(authoritative);
/* Get number of bytes, ignoring trailing spaces */
@@ -2274,14 +2253,10 @@ varstr_abbrev_convert(Datum original, SortSupport ssup)
* thing: explicitly consider string length.
*/
if (sss->collate_c)
- memcpy(pres, authoritative_data, Min(len, sizeof(Datum)));
+ memcpy(pres, authoritative_data, Min(len, max_prefix_bytes));
else
{
Size bsize;
-#ifdef USE_ICU
- int32_t ulen = -1;
- UChar *uchar = NULL;
-#endif
/*
* We're not using the C collation, so fall back on strxfrm or ICU
@@ -2299,7 +2274,7 @@ varstr_abbrev_convert(Datum original, SortSupport ssup)
if (sss->last_len1 == len && sss->cache_blob &&
memcmp(sss->buf1, authoritative_data, len) == 0)
{
- memcpy(pres, sss->buf2, Min(sizeof(Datum), sss->last_len2));
+ memcpy(pres, sss->buf2, Min(max_prefix_bytes, sss->last_len2));
/* No change affecting cardinality, so no hashing required */
goto done;
}
@@ -2307,81 +2282,49 @@ varstr_abbrev_convert(Datum original, SortSupport ssup)
memcpy(sss->buf1, authoritative_data, len);
/*
- * Just like strcoll(), strxfrm() expects a NUL-terminated string. Not
- * necessary for ICU, but doesn't hurt.
+ * pg_strxfrm() and pg_strxfrm_prefix expect NUL-terminated
+ * strings.
*/
sss->buf1[len] = '\0';
sss->last_len1 = len;
-#ifdef USE_ICU
- /* When using ICU and not UTF8, convert string to UChar. */
- if (sss->locale && sss->locale->provider == COLLPROVIDER_ICU &&
- GetDatabaseEncoding() != PG_UTF8)
- ulen = icu_to_uchar(&uchar, sss->buf1, len);
-#endif
-
- /*
- * Loop: Call strxfrm() or ucol_getSortKey(), possibly enlarge buffer,
- * and try again. Both of these functions have the result buffer
- * content undefined if the result did not fit, so we need to retry
- * until everything fits, even though we only need the first few bytes
- * in the end. When using ucol_nextSortKeyPart(), however, we only
- * ask for as many bytes as we actually need.
- */
- for (;;)
+ if (pg_strxfrm_prefix_enabled(sss->locale))
{
-#ifdef USE_ICU
- if (sss->locale && sss->locale->provider == COLLPROVIDER_ICU)
+ if (sss->buflen2 < max_prefix_bytes)
{
- /*
- * When using UTF8, use the iteration interface so we only
- * need to produce as many bytes as we actually need.
- */
- if (GetDatabaseEncoding() == PG_UTF8)
- {
- UCharIterator iter;
- uint32_t state[2];
- UErrorCode status;
-
- uiter_setUTF8(&iter, sss->buf1, len);
- state[0] = state[1] = 0; /* won't need that again */
- status = U_ZERO_ERROR;
- bsize = ucol_nextSortKeyPart(sss->locale->info.icu.ucol,
- &iter,
- state,
- (uint8_t *) sss->buf2,
- Min(sizeof(Datum), sss->buflen2),
- &status);
- if (U_FAILURE(status))
- ereport(ERROR,
- (errmsg("sort key generation failed: %s",
- u_errorName(status))));
- }
- else
- bsize = ucol_getSortKey(sss->locale->info.icu.ucol,
- uchar, ulen,
- (uint8_t *) sss->buf2, sss->buflen2);
+ sss->buflen2 = Max(max_prefix_bytes,
+ Min(sss->buflen2 * 2, MaxAllocSize));
+ sss->buf2 = repalloc(sss->buf2, sss->buflen2);
}
- else
-#endif
-#ifdef HAVE_LOCALE_T
- if (sss->locale && sss->locale->provider == COLLPROVIDER_LIBC)
- bsize = strxfrm_l(sss->buf2, sss->buf1,
- sss->buflen2, sss->locale->info.lt);
- else
-#endif
- bsize = strxfrm(sss->buf2, sss->buf1, sss->buflen2);
-
- sss->last_len2 = bsize;
- if (bsize < sss->buflen2)
- break;
+ bsize = pg_strxfrm_prefix(sss->buf2, sss->buf1,
+ max_prefix_bytes, sss->locale);
+ }
+ else
+ {
/*
- * Grow buffer and retry.
+ * Loop: Call pg_strxfrm(), possibly enlarge buffer, and try
+ * again. The pg_strxfrm() function leaves the result buffer
+ * content undefined if the result did not fit, so we need to
+ * retry until everything fits, even though we only need the first
+ * few bytes in the end.
*/
- sss->buflen2 = Max(bsize + 1,
- Min(sss->buflen2 * 2, MaxAllocSize));
- sss->buf2 = repalloc(sss->buf2, sss->buflen2);
+ for (;;)
+ {
+ bsize = pg_strxfrm(sss->buf2, sss->buf1, sss->buflen2,
+ sss->locale);
+
+ sss->last_len2 = bsize;
+ if (bsize < sss->buflen2)
+ break;
+
+ /*
+ * Grow buffer and retry.
+ */
+ sss->buflen2 = Max(bsize + 1,
+ Min(sss->buflen2 * 2, MaxAllocSize));
+ sss->buf2 = repalloc(sss->buf2, sss->buflen2);
+ }
}
/*
@@ -2393,12 +2336,7 @@ varstr_abbrev_convert(Datum original, SortSupport ssup)
* (Actually, even if there were NUL bytes in the blob it would be
* okay. See remarks on bytea case above.)
*/
- memcpy(pres, sss->buf2, Min(sizeof(Datum), bsize));
-
-#ifdef USE_ICU
- if (uchar)
- pfree(uchar);
-#endif
+ memcpy(pres, sss->buf2, Min(max_prefix_bytes, bsize));
}
/*
diff --git a/src/include/utils/pg_locale.h b/src/include/utils/pg_locale.h
index bf70ae08ca..ceab0d4307 100644
--- a/src/include/utils/pg_locale.h
+++ b/src/include/utils/pg_locale.h
@@ -103,6 +103,16 @@ extern char *get_collation_actual_version(char collprovider, const char *collcol
extern int pg_strcoll(const char *arg1, const char *arg2, pg_locale_t locale);
extern int pg_strncoll(const char *arg1, size_t len1,
const char *arg2, size_t len2, pg_locale_t locale);
+extern bool pg_strxfrm_enabled(pg_locale_t locale);
+extern size_t pg_strxfrm(char *dest, const char *src, size_t destsize,
+ pg_locale_t locale);
+extern size_t pg_strnxfrm(char *dest, size_t destsize, const char *src,
+ size_t srclen, pg_locale_t locale);
+extern bool pg_strxfrm_prefix_enabled(pg_locale_t locale);
+extern size_t pg_strxfrm_prefix(char *dest, const char *src, size_t destsize,
+ pg_locale_t locale);
+extern size_t pg_strnxfrm_prefix(char *dest, size_t destsize, const char *src,
+ size_t srclen, pg_locale_t locale);
#ifdef USE_ICU
extern int32_t icu_to_uchar(UChar **buff_uchar, const char *buff, size_t nbytes);
--
2.34.1
[text/x-patch] v2-0003-Refactor-pg_locale_t-routines.patch (45.7K, ../[email protected]/4-v2-0003-Refactor-pg_locale_t-routines.patch)
download | inline diff:
From 0145d9050ba3feef163c1d4a13660f904167458a Mon Sep 17 00:00:00 2001
From: Jeff Davis <[email protected]>
Date: Mon, 5 Dec 2022 10:43:52 -0800
Subject: [PATCH v2 3/6] Refactor pg_locale_t routines.
* add pg_locale_internal.h to hide pg_locale_struct
* move info.lt into info.libc.lt to match icu
* introduce init_default_locale()
* introduce collation_version_default_locale()
* introduce pg_locale_deterministic() accessor
* make default_locale a static global in pg_locale.c
* refactor pg_newlocale_from_collation() to use multiple static
functions and avoid allocating in TopMemoryContext until
necessary
* refactor get_collation_actual_version() to use
pg_newlocale()
---
src/backend/access/hash/hashfunc.c | 82 ++---
src/backend/commands/collationcmds.c | 1 +
src/backend/regex/regc_pg_locale.c | 45 +--
src/backend/utils/adt/formatting.c | 25 +-
src/backend/utils/adt/like.c | 3 +-
src/backend/utils/adt/like_support.c | 3 +-
src/backend/utils/adt/pg_locale.c | 459 +++++++++++++++++--------
src/backend/utils/adt/varchar.c | 62 ++--
src/backend/utils/adt/varlena.c | 8 +-
src/backend/utils/init/postinit.c | 30 +-
src/include/utils/pg_locale.h | 55 +--
src/include/utils/pg_locale_internal.h | 68 ++++
12 files changed, 508 insertions(+), 333 deletions(-)
create mode 100644 src/include/utils/pg_locale_internal.h
diff --git a/src/backend/access/hash/hashfunc.c b/src/backend/access/hash/hashfunc.c
index b8136e496f..6d9f014c5b 100644
--- a/src/backend/access/hash/hashfunc.c
+++ b/src/backend/access/hash/hashfunc.c
@@ -281,36 +281,28 @@ hashtext(PG_FUNCTION_ARGS)
if (!lc_collate_is_c(collid))
mylocale = pg_newlocale_from_collation(collid);
- if (!mylocale || mylocale->deterministic)
+ if (pg_locale_deterministic(mylocale))
{
result = hash_any((unsigned char *) VARDATA_ANY(key),
VARSIZE_ANY_EXHDR(key));
}
else
{
-#ifdef USE_ICU
- if (mylocale->provider == COLLPROVIDER_ICU)
- {
- Size bsize, rsize;
- char *buf;
- const char *keydata = VARDATA_ANY(key);
- size_t keylen = VARSIZE_ANY_EXHDR(key);
-
- bsize = pg_strnxfrm(NULL, 0, keydata, keylen, mylocale);
- buf = palloc(bsize);
-
- rsize = pg_strnxfrm(buf, bsize, keydata, keylen, mylocale);
- if (rsize != bsize)
- elog(ERROR, "pg_strnxfrm() returned unexpected result");
-
- result = hash_any((uint8_t *) buf, bsize);
-
- pfree(buf);
- }
- else
-#endif
- /* shouldn't happen */
- elog(ERROR, "unsupported collprovider: %c", mylocale->provider);
+ Size bsize, rsize;
+ char *buf;
+ const char *keydata = VARDATA_ANY(key);
+ size_t keylen = VARSIZE_ANY_EXHDR(key);
+
+ bsize = pg_strnxfrm(NULL, 0, keydata, keylen, mylocale);
+ buf = palloc(bsize);
+
+ rsize = pg_strnxfrm(buf, bsize, keydata, keylen, mylocale);
+ if (rsize != bsize)
+ elog(ERROR, "pg_strnxfrm() returned unexpected result");
+
+ result = hash_any((uint8_t *) buf, bsize);
+
+ pfree(buf);
}
/* Avoid leaking memory for toasted inputs */
@@ -336,7 +328,7 @@ hashtextextended(PG_FUNCTION_ARGS)
if (!lc_collate_is_c(collid))
mylocale = pg_newlocale_from_collation(collid);
- if (!mylocale || mylocale->deterministic)
+ if (pg_locale_deterministic(mylocale))
{
result = hash_any_extended((unsigned char *) VARDATA_ANY(key),
VARSIZE_ANY_EXHDR(key),
@@ -344,30 +336,22 @@ hashtextextended(PG_FUNCTION_ARGS)
}
else
{
-#ifdef USE_ICU
- if (mylocale->provider == COLLPROVIDER_ICU)
- {
- Size bsize, rsize;
- char *buf;
- const char *keydata = VARDATA_ANY(key);
- size_t keylen = VARSIZE_ANY_EXHDR(key);
-
- bsize = pg_strnxfrm(NULL, 0, keydata, keylen, mylocale);
- buf = palloc(bsize);
-
- rsize = pg_strnxfrm(buf, bsize, keydata, keylen, mylocale);
- if (rsize != bsize)
- elog(ERROR, "pg_strnxfrm() returned unexpected result");
-
- result = hash_any_extended((uint8_t *) buf, bsize,
- PG_GETARG_INT64(1));
-
- pfree(buf);
- }
- else
-#endif
- /* shouldn't happen */
- elog(ERROR, "unsupported collprovider: %c", mylocale->provider);
+ Size bsize, rsize;
+ char *buf;
+ const char *keydata = VARDATA_ANY(key);
+ size_t keylen = VARSIZE_ANY_EXHDR(key);
+
+ bsize = pg_strnxfrm(NULL, 0, keydata, keylen, mylocale);
+ buf = palloc(bsize);
+
+ rsize = pg_strnxfrm(buf, bsize, keydata, keylen, mylocale);
+ if (rsize != bsize)
+ elog(ERROR, "pg_strnxfrm() returned unexpected result");
+
+ result = hash_any_extended((uint8_t *) buf, bsize,
+ PG_GETARG_INT64(1));
+
+ pfree(buf);
}
PG_FREE_IF_COPY(key, 0);
diff --git a/src/backend/commands/collationcmds.c b/src/backend/commands/collationcmds.c
index 81e54e0ce6..9e84da4891 100644
--- a/src/backend/commands/collationcmds.c
+++ b/src/backend/commands/collationcmds.c
@@ -36,6 +36,7 @@
#include "utils/builtins.h"
#include "utils/lsyscache.h"
#include "utils/pg_locale.h"
+#include "utils/pg_locale_internal.h"
#include "utils/rel.h"
#include "utils/syscache.h"
diff --git a/src/backend/regex/regc_pg_locale.c b/src/backend/regex/regc_pg_locale.c
index 02d462a659..ac05efb558 100644
--- a/src/backend/regex/regc_pg_locale.c
+++ b/src/backend/regex/regc_pg_locale.c
@@ -17,6 +17,7 @@
#include "catalog/pg_collation.h"
#include "utils/pg_locale.h"
+#include "utils/pg_locale_internal.h"
/*
* To provide as much functionality as possible on a variety of platforms,
@@ -306,13 +307,13 @@ pg_wc_isdigit(pg_wchar c)
case PG_REGEX_LOCALE_WIDE_L:
#ifdef HAVE_LOCALE_T
if (sizeof(wchar_t) >= 4 || c <= (pg_wchar) 0xFFFF)
- return iswdigit_l((wint_t) c, pg_regex_locale->info.lt);
+ return iswdigit_l((wint_t) c, pg_regex_locale->info.libc.lt);
#endif
/* FALL THRU */
case PG_REGEX_LOCALE_1BYTE_L:
#ifdef HAVE_LOCALE_T
return (c <= (pg_wchar) UCHAR_MAX &&
- isdigit_l((unsigned char) c, pg_regex_locale->info.lt));
+ isdigit_l((unsigned char) c, pg_regex_locale->info.libc.lt));
#endif
break;
case PG_REGEX_LOCALE_ICU:
@@ -342,13 +343,13 @@ pg_wc_isalpha(pg_wchar c)
case PG_REGEX_LOCALE_WIDE_L:
#ifdef HAVE_LOCALE_T
if (sizeof(wchar_t) >= 4 || c <= (pg_wchar) 0xFFFF)
- return iswalpha_l((wint_t) c, pg_regex_locale->info.lt);
+ return iswalpha_l((wint_t) c, pg_regex_locale->info.libc.lt);
#endif
/* FALL THRU */
case PG_REGEX_LOCALE_1BYTE_L:
#ifdef HAVE_LOCALE_T
return (c <= (pg_wchar) UCHAR_MAX &&
- isalpha_l((unsigned char) c, pg_regex_locale->info.lt));
+ isalpha_l((unsigned char) c, pg_regex_locale->info.libc.lt));
#endif
break;
case PG_REGEX_LOCALE_ICU:
@@ -378,13 +379,13 @@ pg_wc_isalnum(pg_wchar c)
case PG_REGEX_LOCALE_WIDE_L:
#ifdef HAVE_LOCALE_T
if (sizeof(wchar_t) >= 4 || c <= (pg_wchar) 0xFFFF)
- return iswalnum_l((wint_t) c, pg_regex_locale->info.lt);
+ return iswalnum_l((wint_t) c, pg_regex_locale->info.libc.lt);
#endif
/* FALL THRU */
case PG_REGEX_LOCALE_1BYTE_L:
#ifdef HAVE_LOCALE_T
return (c <= (pg_wchar) UCHAR_MAX &&
- isalnum_l((unsigned char) c, pg_regex_locale->info.lt));
+ isalnum_l((unsigned char) c, pg_regex_locale->info.libc.lt));
#endif
break;
case PG_REGEX_LOCALE_ICU:
@@ -423,13 +424,13 @@ pg_wc_isupper(pg_wchar c)
case PG_REGEX_LOCALE_WIDE_L:
#ifdef HAVE_LOCALE_T
if (sizeof(wchar_t) >= 4 || c <= (pg_wchar) 0xFFFF)
- return iswupper_l((wint_t) c, pg_regex_locale->info.lt);
+ return iswupper_l((wint_t) c, pg_regex_locale->info.libc.lt);
#endif
/* FALL THRU */
case PG_REGEX_LOCALE_1BYTE_L:
#ifdef HAVE_LOCALE_T
return (c <= (pg_wchar) UCHAR_MAX &&
- isupper_l((unsigned char) c, pg_regex_locale->info.lt));
+ isupper_l((unsigned char) c, pg_regex_locale->info.libc.lt));
#endif
break;
case PG_REGEX_LOCALE_ICU:
@@ -459,13 +460,13 @@ pg_wc_islower(pg_wchar c)
case PG_REGEX_LOCALE_WIDE_L:
#ifdef HAVE_LOCALE_T
if (sizeof(wchar_t) >= 4 || c <= (pg_wchar) 0xFFFF)
- return iswlower_l((wint_t) c, pg_regex_locale->info.lt);
+ return iswlower_l((wint_t) c, pg_regex_locale->info.libc.lt);
#endif
/* FALL THRU */
case PG_REGEX_LOCALE_1BYTE_L:
#ifdef HAVE_LOCALE_T
return (c <= (pg_wchar) UCHAR_MAX &&
- islower_l((unsigned char) c, pg_regex_locale->info.lt));
+ islower_l((unsigned char) c, pg_regex_locale->info.libc.lt));
#endif
break;
case PG_REGEX_LOCALE_ICU:
@@ -495,13 +496,13 @@ pg_wc_isgraph(pg_wchar c)
case PG_REGEX_LOCALE_WIDE_L:
#ifdef HAVE_LOCALE_T
if (sizeof(wchar_t) >= 4 || c <= (pg_wchar) 0xFFFF)
- return iswgraph_l((wint_t) c, pg_regex_locale->info.lt);
+ return iswgraph_l((wint_t) c, pg_regex_locale->info.libc.lt);
#endif
/* FALL THRU */
case PG_REGEX_LOCALE_1BYTE_L:
#ifdef HAVE_LOCALE_T
return (c <= (pg_wchar) UCHAR_MAX &&
- isgraph_l((unsigned char) c, pg_regex_locale->info.lt));
+ isgraph_l((unsigned char) c, pg_regex_locale->info.libc.lt));
#endif
break;
case PG_REGEX_LOCALE_ICU:
@@ -531,13 +532,13 @@ pg_wc_isprint(pg_wchar c)
case PG_REGEX_LOCALE_WIDE_L:
#ifdef HAVE_LOCALE_T
if (sizeof(wchar_t) >= 4 || c <= (pg_wchar) 0xFFFF)
- return iswprint_l((wint_t) c, pg_regex_locale->info.lt);
+ return iswprint_l((wint_t) c, pg_regex_locale->info.libc.lt);
#endif
/* FALL THRU */
case PG_REGEX_LOCALE_1BYTE_L:
#ifdef HAVE_LOCALE_T
return (c <= (pg_wchar) UCHAR_MAX &&
- isprint_l((unsigned char) c, pg_regex_locale->info.lt));
+ isprint_l((unsigned char) c, pg_regex_locale->info.libc.lt));
#endif
break;
case PG_REGEX_LOCALE_ICU:
@@ -567,13 +568,13 @@ pg_wc_ispunct(pg_wchar c)
case PG_REGEX_LOCALE_WIDE_L:
#ifdef HAVE_LOCALE_T
if (sizeof(wchar_t) >= 4 || c <= (pg_wchar) 0xFFFF)
- return iswpunct_l((wint_t) c, pg_regex_locale->info.lt);
+ return iswpunct_l((wint_t) c, pg_regex_locale->info.libc.lt);
#endif
/* FALL THRU */
case PG_REGEX_LOCALE_1BYTE_L:
#ifdef HAVE_LOCALE_T
return (c <= (pg_wchar) UCHAR_MAX &&
- ispunct_l((unsigned char) c, pg_regex_locale->info.lt));
+ ispunct_l((unsigned char) c, pg_regex_locale->info.libc.lt));
#endif
break;
case PG_REGEX_LOCALE_ICU:
@@ -603,13 +604,13 @@ pg_wc_isspace(pg_wchar c)
case PG_REGEX_LOCALE_WIDE_L:
#ifdef HAVE_LOCALE_T
if (sizeof(wchar_t) >= 4 || c <= (pg_wchar) 0xFFFF)
- return iswspace_l((wint_t) c, pg_regex_locale->info.lt);
+ return iswspace_l((wint_t) c, pg_regex_locale->info.libc.lt);
#endif
/* FALL THRU */
case PG_REGEX_LOCALE_1BYTE_L:
#ifdef HAVE_LOCALE_T
return (c <= (pg_wchar) UCHAR_MAX &&
- isspace_l((unsigned char) c, pg_regex_locale->info.lt));
+ isspace_l((unsigned char) c, pg_regex_locale->info.libc.lt));
#endif
break;
case PG_REGEX_LOCALE_ICU:
@@ -647,13 +648,13 @@ pg_wc_toupper(pg_wchar c)
case PG_REGEX_LOCALE_WIDE_L:
#ifdef HAVE_LOCALE_T
if (sizeof(wchar_t) >= 4 || c <= (pg_wchar) 0xFFFF)
- return towupper_l((wint_t) c, pg_regex_locale->info.lt);
+ return towupper_l((wint_t) c, pg_regex_locale->info.libc.lt);
#endif
/* FALL THRU */
case PG_REGEX_LOCALE_1BYTE_L:
#ifdef HAVE_LOCALE_T
if (c <= (pg_wchar) UCHAR_MAX)
- return toupper_l((unsigned char) c, pg_regex_locale->info.lt);
+ return toupper_l((unsigned char) c, pg_regex_locale->info.libc.lt);
#endif
return c;
case PG_REGEX_LOCALE_ICU:
@@ -691,13 +692,13 @@ pg_wc_tolower(pg_wchar c)
case PG_REGEX_LOCALE_WIDE_L:
#ifdef HAVE_LOCALE_T
if (sizeof(wchar_t) >= 4 || c <= (pg_wchar) 0xFFFF)
- return towlower_l((wint_t) c, pg_regex_locale->info.lt);
+ return towlower_l((wint_t) c, pg_regex_locale->info.libc.lt);
#endif
/* FALL THRU */
case PG_REGEX_LOCALE_1BYTE_L:
#ifdef HAVE_LOCALE_T
if (c <= (pg_wchar) UCHAR_MAX)
- return tolower_l((unsigned char) c, pg_regex_locale->info.lt);
+ return tolower_l((unsigned char) c, pg_regex_locale->info.libc.lt);
#endif
return c;
case PG_REGEX_LOCALE_ICU:
diff --git a/src/backend/utils/adt/formatting.c b/src/backend/utils/adt/formatting.c
index 26f498b5df..a4bc7fa5f5 100644
--- a/src/backend/utils/adt/formatting.c
+++ b/src/backend/utils/adt/formatting.c
@@ -87,6 +87,7 @@
#include "utils/memutils.h"
#include "utils/numeric.h"
#include "utils/pg_locale.h"
+#include "utils/pg_locale_internal.h"
/* ----------
* Convenience macros for error handling
@@ -1611,7 +1612,7 @@ icu_convert_case(ICU_Convert_Func func, pg_locale_t mylocale,
*buff_dest = palloc(len_dest * sizeof(**buff_dest));
status = U_ZERO_ERROR;
len_dest = func(*buff_dest, len_dest, buff_source, len_source,
- mylocale->info.icu.locale, &status);
+ mylocale->ctype, &status);
if (status == U_BUFFER_OVERFLOW_ERROR)
{
/* try again with adjusted length */
@@ -1619,7 +1620,7 @@ icu_convert_case(ICU_Convert_Func func, pg_locale_t mylocale,
*buff_dest = palloc(len_dest * sizeof(**buff_dest));
status = U_ZERO_ERROR;
len_dest = func(*buff_dest, len_dest, buff_source, len_source,
- mylocale->info.icu.locale, &status);
+ mylocale->ctype, &status);
}
if (U_FAILURE(status))
ereport(ERROR,
@@ -1732,7 +1733,7 @@ str_tolower(const char *buff, size_t nbytes, Oid collid)
{
#ifdef HAVE_LOCALE_T
if (mylocale)
- workspace[curr_char] = towlower_l(workspace[curr_char], mylocale->info.lt);
+ workspace[curr_char] = towlower_l(workspace[curr_char], mylocale->info.libc.lt);
else
#endif
workspace[curr_char] = towlower(workspace[curr_char]);
@@ -1765,7 +1766,7 @@ str_tolower(const char *buff, size_t nbytes, Oid collid)
{
#ifdef HAVE_LOCALE_T
if (mylocale)
- *p = tolower_l((unsigned char) *p, mylocale->info.lt);
+ *p = tolower_l((unsigned char) *p, mylocale->info.libc.lt);
else
#endif
*p = pg_tolower((unsigned char) *p);
@@ -1854,7 +1855,7 @@ str_toupper(const char *buff, size_t nbytes, Oid collid)
{
#ifdef HAVE_LOCALE_T
if (mylocale)
- workspace[curr_char] = towupper_l(workspace[curr_char], mylocale->info.lt);
+ workspace[curr_char] = towupper_l(workspace[curr_char], mylocale->info.libc.lt);
else
#endif
workspace[curr_char] = towupper(workspace[curr_char]);
@@ -1887,7 +1888,7 @@ str_toupper(const char *buff, size_t nbytes, Oid collid)
{
#ifdef HAVE_LOCALE_T
if (mylocale)
- *p = toupper_l((unsigned char) *p, mylocale->info.lt);
+ *p = toupper_l((unsigned char) *p, mylocale->info.libc.lt);
else
#endif
*p = pg_toupper((unsigned char) *p);
@@ -1979,10 +1980,10 @@ str_initcap(const char *buff, size_t nbytes, Oid collid)
if (mylocale)
{
if (wasalnum)
- workspace[curr_char] = towlower_l(workspace[curr_char], mylocale->info.lt);
+ workspace[curr_char] = towlower_l(workspace[curr_char], mylocale->info.libc.lt);
else
- workspace[curr_char] = towupper_l(workspace[curr_char], mylocale->info.lt);
- wasalnum = iswalnum_l(workspace[curr_char], mylocale->info.lt);
+ workspace[curr_char] = towupper_l(workspace[curr_char], mylocale->info.libc.lt);
+ wasalnum = iswalnum_l(workspace[curr_char], mylocale->info.libc.lt);
}
else
#endif
@@ -2024,10 +2025,10 @@ str_initcap(const char *buff, size_t nbytes, Oid collid)
if (mylocale)
{
if (wasalnum)
- *p = tolower_l((unsigned char) *p, mylocale->info.lt);
+ *p = tolower_l((unsigned char) *p, mylocale->info.libc.lt);
else
- *p = toupper_l((unsigned char) *p, mylocale->info.lt);
- wasalnum = isalnum_l((unsigned char) *p, mylocale->info.lt);
+ *p = toupper_l((unsigned char) *p, mylocale->info.libc.lt);
+ wasalnum = isalnum_l((unsigned char) *p, mylocale->info.libc.lt);
}
else
#endif
diff --git a/src/backend/utils/adt/like.c b/src/backend/utils/adt/like.c
index 8e671b9fab..98714a0492 100644
--- a/src/backend/utils/adt/like.c
+++ b/src/backend/utils/adt/like.c
@@ -24,6 +24,7 @@
#include "miscadmin.h"
#include "utils/builtins.h"
#include "utils/pg_locale.h"
+#include "utils/pg_locale_internal.h"
#define LIKE_TRUE 1
@@ -96,7 +97,7 @@ SB_lower_char(unsigned char c, pg_locale_t locale, bool locale_is_c)
return pg_ascii_tolower(c);
#ifdef HAVE_LOCALE_T
else if (locale)
- return tolower_l(c, locale->info.lt);
+ return tolower_l(c, locale->info.libc.lt);
#endif
else
return pg_tolower(c);
diff --git a/src/backend/utils/adt/like_support.c b/src/backend/utils/adt/like_support.c
index 2d3aaaaf6b..28d23ac3ab 100644
--- a/src/backend/utils/adt/like_support.c
+++ b/src/backend/utils/adt/like_support.c
@@ -52,6 +52,7 @@
#include "utils/datum.h"
#include "utils/lsyscache.h"
#include "utils/pg_locale.h"
+#include "utils/pg_locale_internal.h"
#include "utils/selfuncs.h"
#include "utils/varlena.h"
@@ -1511,7 +1512,7 @@ pattern_char_isalpha(char c, bool is_multibyte,
(c >= 'A' && c <= 'Z') || (c >= 'a' && c <= 'z');
#ifdef HAVE_LOCALE_T
else if (locale && locale->provider == COLLPROVIDER_LIBC)
- return isalpha_l((unsigned char) c, locale->info.lt);
+ return isalpha_l((unsigned char) c, locale->info.libc.lt);
#endif
else
return isalpha((unsigned char) c);
diff --git a/src/backend/utils/adt/pg_locale.c b/src/backend/utils/adt/pg_locale.c
index 133bb03a13..0a19845df4 100644
--- a/src/backend/utils/adt/pg_locale.c
+++ b/src/backend/utils/adt/pg_locale.c
@@ -65,6 +65,7 @@
#include "utils/lsyscache.h"
#include "utils/memutils.h"
#include "utils/pg_locale.h"
+#include "utils/pg_locale_internal.h"
#include "utils/syscache.h"
#ifdef USE_ICU
@@ -128,6 +129,11 @@ static HTAB *collation_cache = NULL;
static char *IsoLocaleName(const char *);
#endif
+/*
+ * Database default locale.
+ */
+static pg_locale_t default_locale = NULL;
+
#ifdef USE_ICU
/*
* Converter object for converting between ICU's UChar strings and C strings
@@ -1333,7 +1339,7 @@ lc_collate_is_c(Oid collation)
static int result = -1;
char *localeptr;
- if (default_locale.provider == COLLPROVIDER_ICU)
+ if (default_locale->provider == COLLPROVIDER_ICU)
return false;
if (result >= 0)
@@ -1386,7 +1392,7 @@ lc_ctype_is_c(Oid collation)
static int result = -1;
char *localeptr;
- if (default_locale.provider == COLLPROVIDER_ICU)
+ if (default_locale->provider == COLLPROVIDER_ICU)
return false;
if (result >= 0)
@@ -1417,38 +1423,6 @@ lc_ctype_is_c(Oid collation)
return (lookup_collation_cache(collation, true))->ctype_is_c;
}
-struct pg_locale_struct default_locale;
-
-void
-make_icu_collator(const char *iculocstr,
- struct pg_locale_struct *resultp)
-{
-#ifdef USE_ICU
- UCollator *collator;
- UErrorCode status;
-
- status = U_ZERO_ERROR;
- collator = ucol_open(iculocstr, &status);
- if (U_FAILURE(status))
- ereport(ERROR,
- (errmsg("could not open collator for locale \"%s\": %s",
- iculocstr, u_errorName(status))));
-
- if (U_ICU_VERSION_MAJOR_NUM < 54)
- icu_set_collation_attributes(collator, iculocstr);
-
- /* We will leak this string if the caller errors later :-( */
- resultp->info.icu.locale = MemoryContextStrdup(TopMemoryContext, iculocstr);
- resultp->info.icu.ucol = collator;
-#else /* not USE_ICU */
- /* could get here if a collation was created by a build with ICU */
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("ICU is not supported in this build")));
-#endif /* not USE_ICU */
-}
-
-
/* simple subroutine for reporting errors from newlocale() */
#ifdef HAVE_LOCALE_T
static void
@@ -1482,6 +1456,261 @@ report_newlocale_failure(const char *localename)
#endif /* HAVE_LOCALE_T */
+/*
+ * Construct a new pg_locale_t object.
+ *
+ * Passing NULL for the version is allowed; and even if it is specified, the
+ * result may or may not have an exactly matching version. Other parameters
+ * are required. Caller should pass isdefault=true if initializing
+ * default_locale; false otherwise.
+ *
+ * Structures are allocated in CurrentMemoryContext. The libc locale_t or
+ * UCollator is not allocated in any memory context, so the caller should be
+ * sure to call pg_freelocale() to close it.
+ */
+static pg_locale_t
+pg_newlocale(char provider, bool isdefault, bool deterministic,
+ const char *collate, const char *ctype, const char *version)
+{
+ pg_locale_t result = palloc0(sizeof(struct pg_locale_struct));
+
+ /*
+ * If COLLPROVIDER_DEFAULT, caller should use default_locale or NULL
+ * instead.
+ */
+ Assert(provider != COLLPROVIDER_DEFAULT);
+
+ if (provider == COLLPROVIDER_LIBC && isdefault)
+ {
+ /*
+ * When the default locale is libc, the actual locale settings are
+ * controlled by setlocale(), so there's nothing to do here.
+ */
+ }
+ else if (provider == COLLPROVIDER_LIBC)
+ {
+#ifdef HAVE_LOCALE_T
+ locale_t loc;
+
+ /* newlocale's result may be leaked if we encounter an error */
+
+ if (strcmp(collate, ctype) == 0)
+ {
+ /* Normal case where they're the same */
+ errno = 0;
+#ifndef WIN32
+ loc = newlocale(LC_COLLATE_MASK | LC_CTYPE_MASK, collate,
+ NULL);
+#else
+ loc = _create_locale(LC_ALL, collate);
+#endif
+ if (!loc)
+ report_newlocale_failure(collate);
+ }
+ else
+ {
+#ifndef WIN32
+ /* We need two newlocale() steps */
+ locale_t loc1;
+
+ errno = 0;
+ loc1 = newlocale(LC_COLLATE_MASK, collate, NULL);
+ if (!loc1)
+ report_newlocale_failure(collate);
+ errno = 0;
+ loc = newlocale(LC_CTYPE_MASK, ctype, loc1);
+ if (!loc)
+ report_newlocale_failure(ctype);
+#else
+
+ /*
+ * XXX The _create_locale() API doesn't appear to support
+ * this. Could perhaps be worked around by changing
+ * pg_locale_t to contain two separate fields.
+ */
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("collations with different collate and ctype values are not supported on this platform")));
+#endif
+ }
+
+ result->info.libc.lt = loc;
+#else /* not HAVE_LOCALE_T */
+ /* platform that doesn't support locale_t */
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("collation provider LIBC is not supported on this platform")));
+#endif /* not HAVE_LOCALE_T */
+ }
+#ifdef USE_ICU
+ else if (provider == COLLPROVIDER_ICU)
+ {
+ UCollator *collator;
+ UErrorCode status;
+
+ /* collator may be leaked if we encounter an error */
+
+ status = U_ZERO_ERROR;
+ collator = ucol_open(collate, &status);
+ if (U_FAILURE(status))
+ ereport(ERROR,
+ (errmsg("could not open collator for locale \"%s\": %s",
+ collate, u_errorName(status))));
+
+ if (U_ICU_VERSION_MAJOR_NUM < 54)
+ icu_set_collation_attributes(collator, collate);
+
+ result->info.icu.ucol = collator;
+ }
+#endif
+ else
+ /* shouldn't happen */
+ elog(ERROR, "unsupported collprovider: %c", provider);
+
+ result->provider = provider;
+ result->deterministic = deterministic;
+ result->collate = pstrdup(collate);
+ result->ctype = pstrdup(ctype);
+
+ return result;
+}
+
+/*
+ * Move resources from the given locale into the given mcxt, consuming and
+ * freeing the given locale and returning the new one.
+ */
+static pg_locale_t
+pg_movelocale(MemoryContext mcxt, pg_locale_t *plocale)
+{
+ pg_locale_t locale = *plocale;
+ pg_locale_t result;
+
+ Assert(locale != default_locale);
+
+ result = MemoryContextAllocZero(mcxt, sizeof(struct pg_locale_struct));
+
+ result->provider = locale->provider;
+ result->deterministic = locale->deterministic;
+
+ if (locale->provider == COLLPROVIDER_LIBC)
+ {
+#ifdef HAVE_LOCALE_T
+ if (locale->info.libc.lt != NULL)
+ {
+ /* not in a memory context; just reassign the pointer */
+ result->info.libc.lt = locale->info.libc.lt;
+ }
+#endif
+ }
+#ifdef USE_ICU
+ else if (locale->provider == COLLPROVIDER_ICU)
+ {
+ /* not in a memory context; just reassign the pointer */
+ result->info.icu.ucol = locale->info.icu.ucol;
+ }
+#endif
+ else
+ /* shouldn't happen */
+ elog(ERROR, "unsupported collprovider: %c", locale->provider);
+
+ result->collate = MemoryContextStrdup(mcxt, locale->collate);
+ pfree(locale->collate);
+
+ result->ctype = MemoryContextStrdup(mcxt, locale->ctype);
+ pfree(locale->ctype);
+
+ pfree(locale);
+ *plocale = NULL;
+
+ return result;
+}
+
+/*
+ * Free pg_locale_t and close libc locale_t or UCollator.
+ */
+#ifdef USE_ICU
+static void
+pg_freelocale(pg_locale_t locale)
+{
+ if (!locale)
+ return;
+
+ Assert(locale != default_locale);
+
+ pfree(locale->collate);
+ pfree(locale->ctype);
+
+ if (locale->provider == COLLPROVIDER_LIBC)
+ {
+#ifdef HAVE_LOCALE_T
+ if (locale->info.libc.lt != NULL)
+ {
+#ifndef WIN32
+ freelocale(locale->info.libc.lt);
+#else
+ _free_locale(locale->info.libc.lt);
+#endif
+ }
+#endif
+ }
+#ifdef USE_ICU
+ else if (locale->provider == COLLPROVIDER_ICU)
+ {
+ ucol_close(locale->info.icu.ucol);
+ }
+#endif
+ else
+ /* shouldn't happen */
+ elog(ERROR, "unsupported collprovider: %c", locale->provider);
+
+ pfree(locale);
+}
+#endif
+
+/*
+ * Accessor so that callers don't need to include pg_locale_internal.h.
+ */
+bool
+pg_locale_deterministic(pg_locale_t locale)
+{
+ if (locale == NULL)
+ return true;
+ else
+ return locale->deterministic;
+}
+
+/*
+ * Initialize default database locale.
+ */
+void
+init_default_locale(char provider, const char *collate, const char *ctype,
+ const char *version)
+{
+ pg_locale_t temp_locale;
+ bool deterministic;
+
+ /*
+ * Default locale is currently always deterministic. Nondeterministic
+ * locales currently don't support pattern matching, which would break a
+ * lot of things if applied globally.
+ */
+ deterministic = true;
+ temp_locale = pg_newlocale(provider, true, deterministic, collate,
+ ctype, version);
+
+ default_locale = pg_movelocale(TopMemoryContext, &temp_locale);
+}
+
+/*
+ * Return palloc'd version string for the default locale.
+ */
+char *
+default_locale_collation_version()
+{
+ return get_collation_actual_version(default_locale->provider,
+ default_locale->collate);
+}
+
/*
* Create a locale_t from a collation OID. Results are cached for the
* lifetime of the backend. Thus, do not free the result with freelocale().
@@ -1506,8 +1735,8 @@ pg_newlocale_from_collation(Oid collid)
if (collid == DEFAULT_COLLATION_OID)
{
- if (default_locale.provider == COLLPROVIDER_ICU)
- return &default_locale;
+ if (default_locale->provider == COLLPROVIDER_ICU)
+ return default_locale;
else
return (pg_locale_t) 0;
}
@@ -1519,107 +1748,65 @@ pg_newlocale_from_collation(Oid collid)
/* We haven't computed this yet in this session, so do it */
HeapTuple tp;
Form_pg_collation collform;
- struct pg_locale_struct result;
- pg_locale_t resultp;
+ pg_locale_t temp_locale;
+ pg_locale_t perm_locale;
Datum datum;
bool isnull;
+ char *collate;
+ char *ctype;
+ char *collversionstr;
tp = SearchSysCache1(COLLOID, ObjectIdGetDatum(collid));
if (!HeapTupleIsValid(tp))
elog(ERROR, "cache lookup failed for collation %u", collid);
collform = (Form_pg_collation) GETSTRUCT(tp);
- /* We'll fill in the result struct locally before allocating memory */
- memset(&result, 0, sizeof(result));
- result.provider = collform->collprovider;
- result.deterministic = collform->collisdeterministic;
+ datum = SysCacheGetAttr(COLLOID, tp, Anum_pg_collation_collversion,
+ &isnull);
+ if (!isnull)
+ collversionstr = TextDatumGetCString(datum);
+ else
+ collversionstr = NULL;
if (collform->collprovider == COLLPROVIDER_LIBC)
{
-#ifdef HAVE_LOCALE_T
- const char *collcollate;
- const char *collctype pg_attribute_unused();
- locale_t loc;
-
- datum = SysCacheGetAttr(COLLOID, tp, Anum_pg_collation_collcollate, &isnull);
+ datum = SysCacheGetAttr(COLLOID, tp, Anum_pg_collation_collcollate,
+ &isnull);
Assert(!isnull);
- collcollate = TextDatumGetCString(datum);
- datum = SysCacheGetAttr(COLLOID, tp, Anum_pg_collation_collctype, &isnull);
+ collate = TextDatumGetCString(datum);
+ datum = SysCacheGetAttr(COLLOID, tp, Anum_pg_collation_collctype,
+ &isnull);
Assert(!isnull);
- collctype = TextDatumGetCString(datum);
-
- if (strcmp(collcollate, collctype) == 0)
- {
- /* Normal case where they're the same */
- errno = 0;
-#ifndef WIN32
- loc = newlocale(LC_COLLATE_MASK | LC_CTYPE_MASK, collcollate,
- NULL);
-#else
- loc = _create_locale(LC_ALL, collcollate);
-#endif
- if (!loc)
- report_newlocale_failure(collcollate);
- }
- else
- {
-#ifndef WIN32
- /* We need two newlocale() steps */
- locale_t loc1;
-
- errno = 0;
- loc1 = newlocale(LC_COLLATE_MASK, collcollate, NULL);
- if (!loc1)
- report_newlocale_failure(collcollate);
- errno = 0;
- loc = newlocale(LC_CTYPE_MASK, collctype, loc1);
- if (!loc)
- report_newlocale_failure(collctype);
-#else
-
- /*
- * XXX The _create_locale() API doesn't appear to support
- * this. Could perhaps be worked around by changing
- * pg_locale_t to contain two separate fields.
- */
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("collations with different collate and ctype values are not supported on this platform")));
-#endif
- }
-
- result.info.lt = loc;
-#else /* not HAVE_LOCALE_T */
- /* platform that doesn't support locale_t */
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("collation provider LIBC is not supported on this platform")));
-#endif /* not HAVE_LOCALE_T */
+ ctype = TextDatumGetCString(datum);
}
+#ifdef USE_ICU
else if (collform->collprovider == COLLPROVIDER_ICU)
{
- const char *iculocstr;
-
- datum = SysCacheGetAttr(COLLOID, tp, Anum_pg_collation_colliculocale, &isnull);
+ datum = SysCacheGetAttr(COLLOID, tp, Anum_pg_collation_colliculocale,
+ &isnull);
Assert(!isnull);
- iculocstr = TextDatumGetCString(datum);
- make_icu_collator(iculocstr, &result);
+ collate = TextDatumGetCString(datum);
+
+ /* for ICU, collate and ctype are both set from iculocale */
+ ctype = collate;
}
+#endif
+ else
+ /* shouldn't happen */
+ elog(ERROR, "unsupported collprovider: %c", collform->collprovider);
- datum = SysCacheGetAttr(COLLOID, tp, Anum_pg_collation_collversion,
- &isnull);
- if (!isnull)
+ temp_locale = pg_newlocale(collform->collprovider, false,
+ collform->collisdeterministic,
+ collate, ctype, collversionstr);
+
+ ReleaseSysCache(tp);
+
+ if (collversionstr != NULL)
{
char *actual_versionstr;
- char *collversionstr;
-
- collversionstr = TextDatumGetCString(datum);
- datum = SysCacheGetAttr(COLLOID, tp, collform->collprovider == COLLPROVIDER_ICU ? Anum_pg_collation_colliculocale : Anum_pg_collation_collcollate, &isnull);
- Assert(!isnull);
+ actual_versionstr = get_collation_actual_version(collform->collprovider, collate);
- actual_versionstr = get_collation_actual_version(collform->collprovider,
- TextDatumGetCString(datum));
if (!actual_versionstr)
{
/*
@@ -1646,13 +1833,10 @@ pg_newlocale_from_collation(Oid collid)
NameStr(collform->collname)))));
}
- ReleaseSysCache(tp);
-
- /* We'll keep the pg_locale_t structures in TopMemoryContext */
- resultp = MemoryContextAlloc(TopMemoryContext, sizeof(*resultp));
- *resultp = result;
+ /* move into TopMemoryContext */
+ perm_locale = pg_movelocale(TopMemoryContext, &temp_locale);
- cache_entry->locale = resultp;
+ cache_entry->locale = perm_locale;
}
return cache_entry->locale;
@@ -1812,7 +1996,7 @@ pg_strncoll_libc_win32_utf8(const char *arg1, size_t len1, const char *arg2,
errno = 0;
#ifdef HAVE_LOCALE_T
if (locale)
- result = wcscoll_l((LPWSTR) a1p, (LPWSTR) a2p, locale->info.lt);
+ result = wcscoll_l((LPWSTR) a1p, (LPWSTR) a2p, locale->info.libc.lt);
else
#endif
result = wcscoll((LPWSTR) a1p, (LPWSTR) a2p);
@@ -1855,7 +2039,7 @@ pg_strcoll_libc(const char *arg1, const char *arg2, pg_locale_t locale)
if (locale)
{
#ifdef HAVE_LOCALE_T
- result = strcoll_l(arg1, arg2, locale->info.lt);
+ result = strcoll_l(arg1, arg2, locale->info.libc.lt);
#else
/* shouldn't happen */
elog(ERROR, "unsupported collprovider: %c", locale->provider);
@@ -2108,7 +2292,7 @@ pg_strxfrm_libc(char *dest, const char *src, size_t destsize,
#ifdef TRUST_STXFRM
#ifdef HAVE_LOCALE_T
if (locale)
- return strxfrm_l(dest, src, destsize, locale->info.lt);
+ return strxfrm_l(dest, src, destsize, locale->info.libc.lt);
else
#endif
return strxfrm(dest, src, destsize);
@@ -2718,19 +2902,16 @@ void
check_icu_locale(const char *icu_locale)
{
#ifdef USE_ICU
- UCollator *collator;
- UErrorCode status;
+ pg_locale_t locale;
- status = U_ZERO_ERROR;
- collator = ucol_open(icu_locale, &status);
- if (U_FAILURE(status))
- ereport(ERROR,
- (errmsg("could not open collator for locale \"%s\": %s",
- icu_locale, u_errorName(status))));
-
- if (U_ICU_VERSION_MAJOR_NUM < 54)
- icu_set_collation_attributes(collator, icu_locale);
- ucol_close(collator);
+ /*
+ * Whether it's deterministic doesn't matter in this case, because it
+ * doesn't affect whether the locale is valid or not; and we're going to
+ * discard the locale anyway.
+ */
+ locale = pg_newlocale(COLLPROVIDER_ICU, false, true, icu_locale,
+ icu_locale, NULL);
+ pg_freelocale(locale);
#else
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
@@ -2793,10 +2974,10 @@ wchar2char(char *to, const wchar_t *from, size_t tolen, pg_locale_t locale)
#ifdef HAVE_LOCALE_T
#ifdef HAVE_WCSTOMBS_L
/* Use wcstombs_l for nondefault locales */
- result = wcstombs_l(to, from, tolen, locale->info.lt);
+ result = wcstombs_l(to, from, tolen, locale->info.libc.lt);
#else /* !HAVE_WCSTOMBS_L */
/* We have to temporarily set the locale as current ... ugh */
- locale_t save_locale = uselocale(locale->info.lt);
+ locale_t save_locale = uselocale(locale->info.libc.lt);
result = wcstombs(to, from, tolen);
@@ -2870,10 +3051,10 @@ char2wchar(wchar_t *to, size_t tolen, const char *from, size_t fromlen,
#ifdef HAVE_LOCALE_T
#ifdef HAVE_MBSTOWCS_L
/* Use mbstowcs_l for nondefault locales */
- result = mbstowcs_l(to, str, tolen, locale->info.lt);
+ result = mbstowcs_l(to, str, tolen, locale->info.libc.lt);
#else /* !HAVE_MBSTOWCS_L */
/* We have to temporarily set the locale as current ... ugh */
- locale_t save_locale = uselocale(locale->info.lt);
+ locale_t save_locale = uselocale(locale->info.libc.lt);
result = mbstowcs(to, str, tolen);
diff --git a/src/backend/utils/adt/varchar.c b/src/backend/utils/adt/varchar.c
index d0bc528e9f..52f27d483d 100644
--- a/src/backend/utils/adt/varchar.c
+++ b/src/backend/utils/adt/varchar.c
@@ -757,7 +757,7 @@ bpchareq(PG_FUNCTION_ARGS)
else
mylocale = pg_newlocale_from_collation(collid);
- if (locale_is_c || !mylocale || mylocale->deterministic)
+ if (locale_is_c || pg_locale_deterministic(mylocale))
{
/*
* Since we only care about equality or not-equality, we can avoid all
@@ -802,7 +802,7 @@ bpcharne(PG_FUNCTION_ARGS)
else
mylocale = pg_newlocale_from_collation(collid);
- if (locale_is_c || !mylocale || mylocale->deterministic)
+ if (locale_is_c || pg_locale_deterministic(mylocale))
{
/*
* Since we only care about equality or not-equality, we can avoid all
@@ -1010,33 +1010,25 @@ hashbpchar(PG_FUNCTION_ARGS)
if (!lc_collate_is_c(collid))
mylocale = pg_newlocale_from_collation(collid);
- if (!mylocale || mylocale->deterministic)
+ if (pg_locale_deterministic(mylocale))
{
result = hash_any((unsigned char *) keydata, keylen);
}
else
{
-#ifdef USE_ICU
- if (mylocale->provider == COLLPROVIDER_ICU)
- {
- Size bsize, rsize;
- char *buf;
+ Size bsize, rsize;
+ char *buf;
- bsize = pg_strnxfrm(NULL, 0, keydata, keylen, mylocale);
- buf = palloc(bsize);
+ bsize = pg_strnxfrm(NULL, 0, keydata, keylen, mylocale);
+ buf = palloc(bsize);
- rsize = pg_strnxfrm(buf, bsize, keydata, keylen, mylocale);
- if (rsize != bsize)
- elog(ERROR, "pg_strnxfrm() returned unexpected result");
+ rsize = pg_strnxfrm(buf, bsize, keydata, keylen, mylocale);
+ if (rsize != bsize)
+ elog(ERROR, "pg_strnxfrm() returned unexpected result");
- result = hash_any((uint8_t *) buf, bsize);
+ result = hash_any((uint8_t *) buf, bsize);
- pfree(buf);
- }
- else
-#endif
- /* shouldn't happen */
- elog(ERROR, "unsupported collprovider: %c", mylocale->provider);
+ pfree(buf);
}
/* Avoid leaking memory for toasted inputs */
@@ -1067,35 +1059,27 @@ hashbpcharextended(PG_FUNCTION_ARGS)
if (!lc_collate_is_c(collid))
mylocale = pg_newlocale_from_collation(collid);
- if (!mylocale || mylocale->deterministic)
+ if (pg_locale_deterministic(mylocale))
{
result = hash_any_extended((unsigned char *) keydata, keylen,
PG_GETARG_INT64(1));
}
else
{
-#ifdef USE_ICU
- if (mylocale->provider == COLLPROVIDER_ICU)
- {
- Size bsize, rsize;
- char *buf;
+ Size bsize, rsize;
+ char *buf;
- bsize = pg_strnxfrm(NULL, 0, keydata, keylen, mylocale);
- buf = palloc(bsize);
+ bsize = pg_strnxfrm(NULL, 0, keydata, keylen, mylocale);
+ buf = palloc(bsize);
- rsize = pg_strnxfrm(buf, bsize, keydata, keylen, mylocale);
- if (rsize != bsize)
- elog(ERROR, "pg_strnxfrm() returned unexpected result");
+ rsize = pg_strnxfrm(buf, bsize, keydata, keylen, mylocale);
+ if (rsize != bsize)
+ elog(ERROR, "pg_strnxfrm() returned unexpected result");
- result = hash_any_extended((uint8_t *) buf, bsize,
- PG_GETARG_INT64(1));
+ result = hash_any_extended((uint8_t *) buf, bsize,
+ PG_GETARG_INT64(1));
- pfree(buf);
- }
- else
-#endif
- /* shouldn't happen */
- elog(ERROR, "unsupported collprovider: %c", mylocale->provider);
+ pfree(buf);
}
PG_FREE_IF_COPY(key, 0);
diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c
index 2dfba4b488..a7c39d7afa 100644
--- a/src/backend/utils/adt/varlena.c
+++ b/src/backend/utils/adt/varlena.c
@@ -1203,7 +1203,7 @@ text_position_setup(text *t1, text *t2, Oid collid, TextPositionState *state)
if (!lc_collate_is_c(collid))
mylocale = pg_newlocale_from_collation(collid);
- if (mylocale && !mylocale->deterministic)
+ if (!pg_locale_deterministic(mylocale))
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg("nondeterministic collations are not supported for substring searches")));
@@ -1601,7 +1601,7 @@ texteq(PG_FUNCTION_ARGS)
else
mylocale = pg_newlocale_from_collation(collid);
- if (locale_is_c || !mylocale || mylocale->deterministic)
+ if (locale_is_c || pg_locale_deterministic(mylocale))
{
Datum arg1 = PG_GETARG_DATUM(0);
Datum arg2 = PG_GETARG_DATUM(1);
@@ -1660,7 +1660,7 @@ textne(PG_FUNCTION_ARGS)
else
mylocale = pg_newlocale_from_collation(collid);
- if (locale_is_c || !mylocale || mylocale->deterministic)
+ if (locale_is_c || pg_locale_deterministic(mylocale))
{
Datum arg1 = PG_GETARG_DATUM(0);
Datum arg2 = PG_GETARG_DATUM(1);
@@ -1774,7 +1774,7 @@ text_starts_with(PG_FUNCTION_ARGS)
if (!lc_collate_is_c(collid))
mylocale = pg_newlocale_from_collation(collid);
- if (mylocale && !mylocale->deterministic)
+ if (!pg_locale_deterministic(mylocale))
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg("nondeterministic collations are not supported for substring searches")));
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index a990c833c5..c5528cbf64 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -317,6 +317,7 @@ CheckMyDatabase(const char *name, bool am_superuser, bool override_allow_connect
char *collate;
char *ctype;
char *iculocale;
+ char *collversionstr;
/* Fetch our pg_database row normally, via syscache */
tup = SearchSysCache1(DATABASEOID, ObjectIdGetDatum(MyDatabaseId));
@@ -424,35 +425,32 @@ CheckMyDatabase(const char *name, bool am_superuser, bool override_allow_connect
datum = SysCacheGetAttr(DATABASEOID, tup, Anum_pg_database_daticulocale, &isnull);
Assert(!isnull);
iculocale = TextDatumGetCString(datum);
- make_icu_collator(iculocale, &default_locale);
}
else
iculocale = NULL;
- default_locale.provider = dbform->datlocprovider;
+ datum = SysCacheGetAttr(DATABASEOID, tup, Anum_pg_database_datcollversion,
+ &isnull);
+ if (!isnull)
+ collversionstr = TextDatumGetCString(datum);
+ else
+ collversionstr = NULL;
- /*
- * Default locale is currently always deterministic. Nondeterministic
- * locales currently don't support pattern matching, which would break a
- * lot of things if applied globally.
- */
- default_locale.deterministic = true;
+ init_default_locale(dbform->datlocprovider,
+ dbform->datlocprovider == COLLPROVIDER_ICU ? iculocale : collate,
+ dbform->datlocprovider == COLLPROVIDER_ICU ? iculocale : ctype,
+ collversionstr);
/*
* Check collation version. See similar code in
* pg_newlocale_from_collation(). Note that here we warn instead of error
* in any case, so that we don't prevent connecting.
*/
- datum = SysCacheGetAttr(DATABASEOID, tup, Anum_pg_database_datcollversion,
- &isnull);
- if (!isnull)
+ if (collversionstr != NULL)
{
char *actual_versionstr;
- char *collversionstr;
- collversionstr = TextDatumGetCString(datum);
-
- actual_versionstr = get_collation_actual_version(dbform->datlocprovider, dbform->datlocprovider == COLLPROVIDER_ICU ? iculocale : collate);
+ actual_versionstr = default_locale_collation_version();
if (!actual_versionstr)
/* should not happen */
elog(WARNING,
@@ -470,6 +468,8 @@ CheckMyDatabase(const char *name, bool am_superuser, bool override_allow_connect
"or build PostgreSQL with the right library version.",
quote_identifier(name))));
}
+ else
+ collversionstr = NULL;
/* Make the locale settings visible as GUC variables, too */
SetConfigOption("lc_collate", collate, PGC_INTERNAL, PGC_S_DYNAMIC_DEFAULT);
diff --git a/src/include/utils/pg_locale.h b/src/include/utils/pg_locale.h
index ceab0d4307..0d7bc0534f 100644
--- a/src/include/utils/pg_locale.h
+++ b/src/include/utils/pg_locale.h
@@ -15,22 +15,6 @@
#if defined(LOCALE_T_IN_XLOCALE) || defined(WCSTOMBS_L_IN_XLOCALE)
#include <xlocale.h>
#endif
-#ifdef USE_ICU
-#include <unicode/ucol.h>
-#endif
-
-#ifdef USE_ICU
-/*
- * ucol_strcollUTF8() was introduced in ICU 50, but it is buggy before ICU 53.
- * (see
- * <https://www.postgresql.org/message-id/flat/f1438ec6-22aa-4029-9a3b-26f79d330e72%40manitou-mail.org>)
- */
-#if U_ICU_VERSION_MAJOR_NUM >= 53
-#define HAVE_UCOL_STRCOLLUTF8 1
-#else
-#undef HAVE_UCOL_STRCOLLUTF8
-#endif
-#endif
/* use for libc locale names */
#define LOCALE_NAME_BUFLEN 128
@@ -64,39 +48,12 @@ extern struct lconv *PGLC_localeconv(void);
extern void cache_locale_time(void);
-/*
- * We define our own wrapper around locale_t so we can keep the same
- * function signatures for all builds, while not having to create a
- * fake version of the standard type locale_t in the global namespace.
- * pg_locale_t is occasionally checked for truth, so make it a pointer.
- */
-struct pg_locale_struct
-{
- char provider;
- bool deterministic;
- union
- {
-#ifdef HAVE_LOCALE_T
- locale_t lt;
-#endif
-#ifdef USE_ICU
- struct
- {
- const char *locale;
- UCollator *ucol;
- } icu;
-#endif
- int dummy; /* in case we have neither LOCALE_T nor ICU */
- } info;
-};
-
typedef struct pg_locale_struct *pg_locale_t;
-extern PGDLLIMPORT struct pg_locale_struct default_locale;
-
-extern void make_icu_collator(const char *iculocstr,
- struct pg_locale_struct *resultp);
-
+extern void init_default_locale(char provider, const char *collate,
+ const char *ctype, const char *version);
+extern char *default_locale_collation_version(void);
+extern bool pg_locale_deterministic(pg_locale_t locale);
extern pg_locale_t pg_newlocale_from_collation(Oid collid);
extern char *get_collation_actual_version(char collprovider, const char *collcollate);
@@ -114,10 +71,6 @@ extern size_t pg_strxfrm_prefix(char *dest, const char *src, size_t destsize,
extern size_t pg_strnxfrm_prefix(char *dest, size_t destsize, const char *src,
size_t srclen, pg_locale_t locale);
-#ifdef USE_ICU
-extern int32_t icu_to_uchar(UChar **buff_uchar, const char *buff, size_t nbytes);
-extern int32_t icu_from_uchar(char **result, const UChar *buff_uchar, int32_t len_uchar);
-#endif
extern void check_icu_locale(const char *icu_locale);
/* These functions convert from/to libc's wchar_t, *not* pg_wchar_t */
diff --git a/src/include/utils/pg_locale_internal.h b/src/include/utils/pg_locale_internal.h
new file mode 100644
index 0000000000..33465ad92d
--- /dev/null
+++ b/src/include/utils/pg_locale_internal.h
@@ -0,0 +1,68 @@
+/*-----------------------------------------------------------------------
+ *
+ * PostgreSQL locale utilities
+ *
+ * src/include/utils/pg_locale_internal.h
+ *
+ * Copyright (c) 2002-2022, PostgreSQL Global Development Group
+ *
+ *-----------------------------------------------------------------------
+ */
+
+
+#ifndef _PG_LOCALE_INTERNAL_
+#define _PG_LOCALE_INTERNAL_
+
+#ifdef USE_ICU
+#include <unicode/ucol.h>
+#endif
+
+#ifdef USE_ICU
+/*
+ * ucol_strcollUTF8() was introduced in ICU 50, but it is buggy before ICU 53.
+ * (see
+ * <https://www.postgresql.org/message-id/flat/f1438ec6-22aa-4029-9a3b-26f79d330e72%40manitou-mail.org>)
+ */
+#if U_ICU_VERSION_MAJOR_NUM >= 53
+#define HAVE_UCOL_STRCOLLUTF8 1
+#else
+#undef HAVE_UCOL_STRCOLLUTF8
+#endif
+#endif
+
+/*
+ * We define our own wrapper around locale_t so we can keep the same
+ * function signatures for all builds, while not having to create a
+ * fake version of the standard type locale_t in the global namespace.
+ * pg_locale_t is occasionally checked for truth, so make it a pointer.
+ */
+struct pg_locale_struct
+{
+ char provider;
+ bool deterministic;
+ char *collate;
+ char *ctype;
+ union
+ {
+#ifdef HAVE_LOCALE_T
+ struct
+ {
+ locale_t lt;
+ } libc;
+#endif
+#ifdef USE_ICU
+ struct
+ {
+ UCollator *ucol;
+ } icu;
+#endif
+ int dummy; /* in case we have neither LOCALE_T nor ICU */
+ } info;
+};
+
+#ifdef USE_ICU
+extern int32_t icu_to_uchar(UChar **buff_uchar, const char *buff, size_t nbytes);
+extern int32_t icu_from_uchar(char **result, const UChar *buff_uchar, int32_t len_uchar);
+#endif
+
+#endif /* _PG_LOCALE_INTERNAL_ */
--
2.34.1
[text/x-patch] v2-0004-Add-method-structure-toward-ICU-multi-library-sup.patch (28.8K, ../[email protected]/5-v2-0004-Add-method-structure-toward-ICU-multi-library-sup.patch)
download | inline diff:
From daef4c5f6ebf6cd73dcae5402feb5ec269a50650 Mon Sep 17 00:00:00 2001
From: Jeff Davis <[email protected]>
Date: Wed, 7 Dec 2022 11:07:31 -0800
Subject: [PATCH v2 4/6] Add method structure, toward ICU multi-library
support.
Introduce structure pg_icu_library, which holds pointers to each
required ICU method, and store this as part of pg_locale_t. Each call
to an ICU function instead goes through this structure, so that it can
more easily be replaced by a non-builtin ICU library.
This is a step toward support for multiple ICU libraries, to allow
control over precisely which version of a library is used and prevent
problems from subtle changes in collation order.
Author: Thomas Munro, Jeff Davis
---
src/backend/commands/collationcmds.c | 17 +-
src/backend/utils/adt/formatting.c | 67 ++++++--
src/backend/utils/adt/pg_locale.c | 227 +++++++++++++++++--------
src/include/utils/pg_locale_internal.h | 113 +++++++++++-
4 files changed, 322 insertions(+), 102 deletions(-)
diff --git a/src/backend/commands/collationcmds.c b/src/backend/commands/collationcmds.c
index 9e84da4891..16b1bcbdc0 100644
--- a/src/backend/commands/collationcmds.c
+++ b/src/backend/commands/collationcmds.c
@@ -560,13 +560,14 @@ get_icu_language_tag(const char *localename)
{
char buf[ULOC_FULLNAME_CAPACITY];
UErrorCode status;
+ pg_icu_library *iculib = get_builtin_icu_library();
status = U_ZERO_ERROR;
- uloc_toLanguageTag(localename, buf, sizeof(buf), true, &status);
+ iculib->toLanguageTag(localename, buf, sizeof(buf), true, &status);
if (U_FAILURE(status))
ereport(ERROR,
(errmsg("could not convert locale name \"%s\" to language tag: %s",
- localename, u_errorName(status))));
+ localename, iculib->errorName(status))));
return pstrdup(buf);
}
@@ -585,11 +586,12 @@ get_icu_locale_comment(const char *localename)
int32 len_uchar;
int32 i;
char *result;
+ pg_icu_library *iculib = get_builtin_icu_library();
status = U_ZERO_ERROR;
- len_uchar = uloc_getDisplayName(localename, "en",
- displayname, lengthof(displayname),
- &status);
+ len_uchar = iculib->getDisplayName(localename, "en",
+ displayname, lengthof(displayname),
+ &status);
if (U_FAILURE(status))
return NULL; /* no good reason to raise an error */
@@ -809,12 +811,13 @@ pg_import_system_collations(PG_FUNCTION_ARGS)
#ifdef USE_ICU
{
int i;
+ pg_icu_library *iculib = get_builtin_icu_library();
/*
* Start the loop at -1 to sneak in the root locale without too much
* code duplication.
*/
- for (i = -1; i < uloc_countAvailable(); i++)
+ for (i = -1; i < iculib->countAvailable(); i++)
{
const char *name;
char *langtag;
@@ -825,7 +828,7 @@ pg_import_system_collations(PG_FUNCTION_ARGS)
if (i == -1)
name = ""; /* ICU root locale */
else
- name = uloc_getAvailable(i);
+ name = iculib->getAvailable(i);
langtag = get_icu_language_tag(name);
iculocstr = U_ICU_VERSION_MAJOR_NUM >= 54 ? langtag : name;
diff --git a/src/backend/utils/adt/formatting.c b/src/backend/utils/adt/formatting.c
index a4bc7fa5f5..289aa569de 100644
--- a/src/backend/utils/adt/formatting.c
+++ b/src/backend/utils/adt/formatting.c
@@ -1600,6 +1600,11 @@ typedef int32_t (*ICU_Convert_Func) (UChar *dest, int32_t destCapacity,
const UChar *src, int32_t srcLength,
const char *locale,
UErrorCode *pErrorCode);
+typedef int32_t (*ICU_Convert_BI_Func) (UChar *dest, int32_t destCapacity,
+ const UChar *src, int32_t srcLength,
+ UBreakIterator *bi,
+ const char *locale,
+ UErrorCode *pErrorCode);
static int32_t
icu_convert_case(ICU_Convert_Func func, pg_locale_t mylocale,
@@ -1607,6 +1612,7 @@ icu_convert_case(ICU_Convert_Func func, pg_locale_t mylocale,
{
UErrorCode status;
int32_t len_dest;
+ pg_icu_library *iculib = PG_ICU_LIB(mylocale);
len_dest = len_source; /* try first with same length */
*buff_dest = palloc(len_dest * sizeof(**buff_dest));
@@ -1624,18 +1630,42 @@ icu_convert_case(ICU_Convert_Func func, pg_locale_t mylocale,
}
if (U_FAILURE(status))
ereport(ERROR,
- (errmsg("case conversion failed: %s", u_errorName(status))));
+ (errmsg("case conversion failed: %s",
+ iculib->errorName(status))));
return len_dest;
}
+/*
+ * Like icu_convert_case, but func takes a break iterator (which we don't
+ * make use of).
+ */
static int32_t
-u_strToTitle_default_BI(UChar *dest, int32_t destCapacity,
- const UChar *src, int32_t srcLength,
- const char *locale,
- UErrorCode *pErrorCode)
+icu_convert_case_bi(ICU_Convert_BI_Func func, pg_locale_t mylocale,
+ UChar **buff_dest, UChar *buff_source, int32_t len_source)
{
- return u_strToTitle(dest, destCapacity, src, srcLength,
- NULL, locale, pErrorCode);
+ UErrorCode status;
+ int32_t len_dest;
+ pg_icu_library *iculib = PG_ICU_LIB(mylocale);
+
+ len_dest = len_source; /* try first with same length */
+ *buff_dest = palloc(len_dest * sizeof(**buff_dest));
+ status = U_ZERO_ERROR;
+ len_dest = func(*buff_dest, len_dest, buff_source, len_source, NULL,
+ mylocale->ctype, &status);
+ if (status == U_BUFFER_OVERFLOW_ERROR)
+ {
+ /* try again with adjusted length */
+ pfree(*buff_dest);
+ *buff_dest = palloc(len_dest * sizeof(**buff_dest));
+ status = U_ZERO_ERROR;
+ len_dest = func(*buff_dest, len_dest, buff_source, len_source, NULL,
+ mylocale->ctype, &status);
+ }
+ if (U_FAILURE(status))
+ ereport(ERROR,
+ (errmsg("case conversion failed: %s",
+ iculib->errorName(status))));
+ return len_dest;
}
#endif /* USE_ICU */
@@ -1701,11 +1731,12 @@ str_tolower(const char *buff, size_t nbytes, Oid collid)
int32_t len_conv;
UChar *buff_uchar;
UChar *buff_conv;
+ pg_icu_library *iculib = PG_ICU_LIB(mylocale);
- len_uchar = icu_to_uchar(&buff_uchar, buff, nbytes);
- len_conv = icu_convert_case(u_strToLower, mylocale,
+ len_uchar = icu_to_uchar(iculib, &buff_uchar, buff, nbytes);
+ len_conv = icu_convert_case(iculib->strToLower, mylocale,
&buff_conv, buff_uchar, len_uchar);
- icu_from_uchar(&result, buff_conv, len_conv);
+ icu_from_uchar(iculib, &result, buff_conv, len_conv);
pfree(buff_uchar);
pfree(buff_conv);
}
@@ -1823,11 +1854,12 @@ str_toupper(const char *buff, size_t nbytes, Oid collid)
len_conv;
UChar *buff_uchar;
UChar *buff_conv;
+ pg_icu_library *iculib = PG_ICU_LIB(mylocale);
- len_uchar = icu_to_uchar(&buff_uchar, buff, nbytes);
- len_conv = icu_convert_case(u_strToUpper, mylocale,
+ len_uchar = icu_to_uchar(iculib, &buff_uchar, buff, nbytes);
+ len_conv = icu_convert_case(iculib->strToUpper, mylocale,
&buff_conv, buff_uchar, len_uchar);
- icu_from_uchar(&result, buff_conv, len_conv);
+ icu_from_uchar(iculib, &result, buff_conv, len_conv);
pfree(buff_uchar);
pfree(buff_conv);
}
@@ -1946,11 +1978,12 @@ str_initcap(const char *buff, size_t nbytes, Oid collid)
len_conv;
UChar *buff_uchar;
UChar *buff_conv;
+ pg_icu_library *iculib = PG_ICU_LIB(mylocale);
- len_uchar = icu_to_uchar(&buff_uchar, buff, nbytes);
- len_conv = icu_convert_case(u_strToTitle_default_BI, mylocale,
- &buff_conv, buff_uchar, len_uchar);
- icu_from_uchar(&result, buff_conv, len_conv);
+ len_uchar = icu_to_uchar(iculib, &buff_uchar, buff, nbytes);
+ len_conv = icu_convert_case_bi(iculib->strToTitle, mylocale,
+ &buff_conv, buff_uchar, len_uchar);
+ icu_from_uchar(iculib, &result, buff_conv, len_conv);
pfree(buff_uchar);
pfree(buff_conv);
}
diff --git a/src/backend/utils/adt/pg_locale.c b/src/backend/utils/adt/pg_locale.c
index 0a19845df4..4daff8b7b5 100644
--- a/src/backend/utils/adt/pg_locale.c
+++ b/src/backend/utils/adt/pg_locale.c
@@ -70,6 +70,8 @@
#ifdef USE_ICU
#include <unicode/ucnv.h>
+#include <unicode/ulocdata.h>
+#include <unicode/ustring.h>
#endif
#ifdef __GLIBC__
@@ -135,6 +137,7 @@ static char *IsoLocaleName(const char *);
static pg_locale_t default_locale = NULL;
#ifdef USE_ICU
+
/*
* Converter object for converting between ICU's UChar strings and C strings
* in database encoding. Since the database encoding doesn't change, we only
@@ -142,13 +145,17 @@ static pg_locale_t default_locale = NULL;
*/
static UConverter *icu_converter = NULL;
-static void init_icu_converter(void);
-static size_t uchar_length(UConverter *converter,
+static void init_icu_converter(pg_icu_library *iculib);
+static size_t uchar_length(pg_icu_library *iculib,
+ UConverter *converter,
const char *str, size_t len);
-static int32_t uchar_convert(UConverter *converter,
+static int32_t uchar_convert(pg_icu_library *iculib,
+ UConverter *converter,
UChar *dest, int32_t destlen,
const char *str, size_t srclen);
-static void icu_set_collation_attributes(UCollator *collator, const char *loc);
+static void icu_set_collation_attributes(pg_icu_library *iculib,
+ UCollator *collator,
+ const char *loc);
#endif
/*
@@ -1455,6 +1462,59 @@ report_newlocale_failure(const char *localename)
}
#endif /* HAVE_LOCALE_T */
+#ifdef USE_ICU
+pg_icu_library *
+get_builtin_icu_library()
+{
+ pg_icu_library *lib;
+
+ /*
+ * These assignments will fail to compile if an incompatible API change is
+ * made to some future version of ICU, at which point we might need to
+ * consider special treatment for different major version ranges, with
+ * intermediate trampoline functions.
+ */
+ lib = palloc0(sizeof(*lib));
+ lib->getICUVersion = u_getVersion;
+ lib->getUnicodeVersion = u_getUnicodeVersion;
+ lib->getCLDRVersion = ulocdata_getCLDRVersion;
+ lib->openCollator = ucol_open;
+ lib->closeCollator = ucol_close;
+ lib->getCollatorVersion = ucol_getVersion;
+ lib->getUCAVersion = ucol_getUCAVersion;
+ lib->versionToString = u_versionToString;
+ lib->strcoll = ucol_strcoll;
+ lib->strcollUTF8 = ucol_strcollUTF8;
+ lib->getSortKey = ucol_getSortKey;
+ lib->nextSortKeyPart = ucol_nextSortKeyPart;
+ lib->setUTF8 = uiter_setUTF8;
+ lib->errorName = u_errorName;
+ lib->strToUpper = u_strToUpper;
+ lib->strToLower = u_strToLower;
+ lib->strToTitle = u_strToTitle;
+ lib->setAttribute = ucol_setAttribute;
+ lib->openConverter = ucnv_open;
+ lib->closeConverter = ucnv_close;
+ lib->fromUChars = ucnv_fromUChars;
+ lib->toUChars = ucnv_toUChars;
+ lib->toLanguageTag = uloc_toLanguageTag;
+ lib->getDisplayName = uloc_getDisplayName;
+ lib->countAvailable = uloc_countAvailable;
+ lib->getAvailable = uloc_getAvailable;
+
+ /*
+ * Also assert the size of a couple of types used as output buffers, as a
+ * canary to tell us to add extra padding in the (unlikely) event that a
+ * later release makes these values smaller.
+ */
+ StaticAssertStmt(U_MAX_VERSION_STRING_LENGTH == 20,
+ "u_versionToString output buffer size changed incompatibly");
+ StaticAssertStmt(U_MAX_VERSION_LENGTH == 4,
+ "ucol_getVersion output buffer size changed incompatibly");
+
+ return lib;
+}
+#endif
/*
* Construct a new pg_locale_t object.
@@ -1547,20 +1607,22 @@ pg_newlocale(char provider, bool isdefault, bool deterministic,
{
UCollator *collator;
UErrorCode status;
+ pg_icu_library *iculib = get_builtin_icu_library();
/* collator may be leaked if we encounter an error */
status = U_ZERO_ERROR;
- collator = ucol_open(collate, &status);
+ collator = iculib->openCollator(collate, &status);
if (U_FAILURE(status))
ereport(ERROR,
(errmsg("could not open collator for locale \"%s\": %s",
- collate, u_errorName(status))));
+ collate, iculib->errorName(status))));
if (U_ICU_VERSION_MAJOR_NUM < 54)
- icu_set_collation_attributes(collator, collate);
+ icu_set_collation_attributes(iculib, collator, collate);
result->info.icu.ucol = collator;
+ result->info.icu.lib = iculib;
}
#endif
else
@@ -1607,6 +1669,10 @@ pg_movelocale(MemoryContext mcxt, pg_locale_t *plocale)
{
/* not in a memory context; just reassign the pointer */
result->info.icu.ucol = locale->info.icu.ucol;
+
+ result->info.icu.lib = MemoryContextAlloc(mcxt, sizeof(pg_icu_library));
+ memcpy(result->info.icu.lib, locale->info.icu.lib, sizeof(pg_icu_library));
+ pfree(locale->info.icu.lib);
}
#endif
else
@@ -1656,7 +1722,9 @@ pg_freelocale(pg_locale_t locale)
#ifdef USE_ICU
else if (locale->provider == COLLPROVIDER_ICU)
{
- ucol_close(locale->info.icu.ucol);
+ pg_icu_library *iculib = PG_ICU_LIB(locale);
+ iculib->closeCollator(locale->info.icu.ucol);
+ pfree(locale->info.icu.lib);
}
#endif
else
@@ -1858,17 +1926,18 @@ get_collation_actual_version(char collprovider, const char *collcollate)
UErrorCode status;
UVersionInfo versioninfo;
char buf[U_MAX_VERSION_STRING_LENGTH];
+ pg_icu_library *iculib = get_builtin_icu_library();
status = U_ZERO_ERROR;
- collator = ucol_open(collcollate, &status);
+ collator = iculib->openCollator(collcollate, &status);
if (U_FAILURE(status))
ereport(ERROR,
(errmsg("could not open collator for locale \"%s\": %s",
- collcollate, u_errorName(status))));
- ucol_getVersion(collator, versioninfo);
- ucol_close(collator);
+ collcollate, iculib->errorName(status))));
+ iculib->getCollatorVersion(collator, versioninfo);
+ iculib->closeCollator(collator);
- u_versionToString(versioninfo, buf);
+ iculib->versionToString(versioninfo, buf);
collversion = pstrdup(buf);
}
else
@@ -2120,16 +2189,17 @@ pg_strncoll_icu_no_utf8(const char *arg1, size_t len1,
UChar *uchar1,
*uchar2;
int result;
+ pg_icu_library *iculib = PG_ICU_LIB(locale);
Assert(locale->provider == COLLPROVIDER_ICU);
#ifdef HAVE_UCOL_STRCOLLUTF8
Assert(GetDatabaseEncoding() != PG_UTF8);
#endif
- init_icu_converter();
+ init_icu_converter(iculib);
- ulen1 = uchar_length(icu_converter, arg1, len1);
- ulen2 = uchar_length(icu_converter, arg2, len2);
+ ulen1 = uchar_length(iculib, icu_converter, arg1, len1);
+ ulen2 = uchar_length(iculib, icu_converter, arg2, len2);
bufsize1 = (ulen1 + 1) * sizeof(UChar);
bufsize2 = (ulen2 + 1) * sizeof(UChar);
@@ -2140,12 +2210,12 @@ pg_strncoll_icu_no_utf8(const char *arg1, size_t len1,
uchar1 = (UChar *) buf;
uchar2 = (UChar *) (buf + bufsize1);
- ulen1 = uchar_convert(icu_converter, uchar1, ulen1 + 1, arg1, len1);
- ulen2 = uchar_convert(icu_converter, uchar2, ulen2 + 1, arg2, len2);
+ ulen1 = uchar_convert(iculib, icu_converter, uchar1, ulen1 + 1, arg1, len1);
+ ulen2 = uchar_convert(iculib, icu_converter, uchar2, ulen2 + 1, arg2, len2);
- result = ucol_strcoll(locale->info.icu.ucol,
- uchar1, ulen1,
- uchar2, ulen2);
+ result = iculib->strcoll(locale->info.icu.ucol,
+ uchar1, ulen1,
+ uchar2, ulen2);
if (buf != sbuf)
pfree(buf);
@@ -2166,6 +2236,7 @@ pg_strncoll_icu(const char *arg1, size_t len1, const char *arg2, size_t len2,
pg_locale_t locale)
{
int result;
+ pg_icu_library *iculib = PG_ICU_LIB(locale);
Assert(locale->provider == COLLPROVIDER_ICU);
@@ -2175,13 +2246,14 @@ pg_strncoll_icu(const char *arg1, size_t len1, const char *arg2, size_t len2,
UErrorCode status;
status = U_ZERO_ERROR;
- result = ucol_strcollUTF8(locale->info.icu.ucol,
- arg1, len1,
- arg2, len2,
- &status);
+ result = iculib->strcollUTF8(locale->info.icu.ucol,
+ arg1, len1,
+ arg2, len2,
+ &status);
if (U_FAILURE(status))
ereport(ERROR,
- (errmsg("collation failed: %s", u_errorName(status))));
+ (errmsg("collation failed: %s",
+ iculib->errorName(status))));
}
else
#endif
@@ -2360,12 +2432,13 @@ pg_strnxfrm_icu(char *dest, const char *src, size_t srclen, size_t destsize,
int32_t ulen;
size_t uchar_bsize;
Size result_bsize;
+ pg_icu_library *iculib = PG_ICU_LIB(locale);
Assert(locale->provider == COLLPROVIDER_ICU);
- init_icu_converter();
+ init_icu_converter(iculib);
- ulen = uchar_length(icu_converter, src, srclen);
+ ulen = uchar_length(iculib, icu_converter, src, srclen);
uchar_bsize = (ulen + 1) * sizeof(UChar);
@@ -2374,11 +2447,11 @@ pg_strnxfrm_icu(char *dest, const char *src, size_t srclen, size_t destsize,
uchar = (UChar *) buf;
- ulen = uchar_convert(icu_converter, uchar, ulen + 1, src, srclen);
+ ulen = uchar_convert(iculib, icu_converter, uchar, ulen + 1, src, srclen);
- result_bsize = ucol_getSortKey(locale->info.icu.ucol,
- uchar, ulen,
- (uint8_t *) dest, destsize);
+ result_bsize = iculib->getSortKey(locale->info.icu.ucol,
+ uchar, ulen,
+ (uint8_t *) dest, destsize);
if (buf != sbuf)
pfree(buf);
@@ -2407,13 +2480,14 @@ pg_strnxfrm_prefix_icu_no_utf8(char *dest, const char *src, size_t srclen,
UChar *uchar = NULL;
size_t uchar_bsize;
Size result_bsize;
+ pg_icu_library *iculib = PG_ICU_LIB(locale);
Assert(locale->provider == COLLPROVIDER_ICU);
Assert(GetDatabaseEncoding() != PG_UTF8);
- init_icu_converter();
+ init_icu_converter(iculib);
- ulen = uchar_length(icu_converter, src, srclen);
+ ulen = uchar_length(iculib, icu_converter, src, srclen);
uchar_bsize = (ulen + 1) * sizeof(UChar);
@@ -2422,21 +2496,19 @@ pg_strnxfrm_prefix_icu_no_utf8(char *dest, const char *src, size_t srclen,
uchar = (UChar *) buf;
- ulen = uchar_convert(icu_converter, uchar, ulen + 1, src, srclen);
+ ulen = uchar_convert(iculib, icu_converter, uchar, ulen + 1, src, srclen);
uiter_setString(&iter, uchar, ulen);
state[0] = state[1] = 0; /* won't need that again */
status = U_ZERO_ERROR;
- result_bsize = ucol_nextSortKeyPart(locale->info.icu.ucol,
- &iter,
- state,
- (uint8_t *) dest,
- destsize,
- &status);
+ result_bsize = iculib->nextSortKeyPart(
+ locale->info.icu.ucol, &iter, state,
+ (uint8_t *) dest, destsize, &status);
+
if (U_FAILURE(status))
ereport(ERROR,
(errmsg("sort key generation failed: %s",
- u_errorName(status))));
+ iculib->errorName(status))));
return result_bsize;
}
@@ -2445,6 +2517,7 @@ static size_t
pg_strnxfrm_prefix_icu(char *dest, const char *src, size_t srclen,
size_t destsize, pg_locale_t locale)
{
+ pg_icu_library *iculib = PG_ICU_LIB(locale);
size_t result;
Assert(locale->provider == COLLPROVIDER_ICU);
@@ -2455,19 +2528,17 @@ pg_strnxfrm_prefix_icu(char *dest, const char *src, size_t srclen,
uint32_t state[2];
UErrorCode status;
- uiter_setUTF8(&iter, src, srclen);
+ iculib->setUTF8(&iter, src, srclen);
state[0] = state[1] = 0; /* won't need that again */
status = U_ZERO_ERROR;
- result = ucol_nextSortKeyPart(locale->info.icu.ucol,
- &iter,
- state,
- (uint8_t *) dest,
- destsize,
- &status);
+ result = iculib->nextSortKeyPart(
+ locale->info.icu.ucol, &iter, state,
+ (uint8_t *) dest, destsize, &status);
+
if (U_FAILURE(status))
ereport(ERROR,
(errmsg("sort key generation failed: %s",
- u_errorName(status))));
+ iculib->errorName(status))));
}
else
result = pg_strnxfrm_prefix_icu_no_utf8(dest, src, srclen, destsize,
@@ -2667,7 +2738,7 @@ pg_strnxfrm_prefix(char *dest, size_t destsize, const char *src,
#ifdef USE_ICU
static void
-init_icu_converter(void)
+init_icu_converter(pg_icu_library *iculib)
{
const char *icu_encoding_name;
UErrorCode status;
@@ -2684,11 +2755,11 @@ init_icu_converter(void)
pg_encoding_to_char(GetDatabaseEncoding()))));
status = U_ZERO_ERROR;
- conv = ucnv_open(icu_encoding_name, &status);
+ conv = iculib->openConverter(icu_encoding_name, &status);
if (U_FAILURE(status))
ereport(ERROR,
(errmsg("could not open ICU converter for encoding \"%s\": %s",
- icu_encoding_name, u_errorName(status))));
+ icu_encoding_name, iculib->errorName(status))));
icu_converter = conv;
}
@@ -2697,14 +2768,15 @@ init_icu_converter(void)
* Find length, in UChars, of given string if converted to UChar string.
*/
static size_t
-uchar_length(UConverter *converter, const char *str, size_t len)
+uchar_length(pg_icu_library *iculib, UConverter *converter, const char *str, size_t len)
{
UErrorCode status = U_ZERO_ERROR;
int32_t ulen;
- ulen = ucnv_toUChars(converter, NULL, 0, str, len, &status);
+ ulen = iculib->toUChars(converter, NULL, 0, str, len, &status);
if (U_FAILURE(status) && status != U_BUFFER_OVERFLOW_ERROR)
ereport(ERROR,
- (errmsg("%s failed: %s", "ucnv_toUChars", u_errorName(status))));
+ (errmsg("%s failed: %s", "ucnv_toUChars",
+ iculib->errorName(status))));
return ulen;
}
@@ -2713,16 +2785,17 @@ uchar_length(UConverter *converter, const char *str, size_t len)
* return the length (in UChars).
*/
static int32_t
-uchar_convert(UConverter *converter, UChar *dest, int32_t destlen,
- const char *src, size_t srclen)
+uchar_convert(pg_icu_library *iculib, UConverter *converter, UChar *dest,
+ int32_t destlen, const char *src, size_t srclen)
{
UErrorCode status = U_ZERO_ERROR;
int32_t ulen;
status = U_ZERO_ERROR;
- ulen = ucnv_toUChars(converter, dest, destlen, src, srclen, &status);
+ ulen = iculib->toUChars(converter, dest, destlen, src, srclen, &status);
if (U_FAILURE(status))
ereport(ERROR,
- (errmsg("%s failed: %s", "ucnv_toUChars", u_errorName(status))));
+ (errmsg("%s failed: %s", "ucnv_toUChars",
+ iculib->errorName(status))));
return ulen;
}
@@ -2739,16 +2812,17 @@ uchar_convert(UConverter *converter, UChar *dest, int32_t destlen,
* result length instead.
*/
int32_t
-icu_to_uchar(UChar **buff_uchar, const char *buff, size_t nbytes)
+icu_to_uchar(pg_icu_library *iculib, UChar **buff_uchar, const char *buff,
+ size_t nbytes)
{
int32_t len_uchar;
- init_icu_converter();
+ init_icu_converter(iculib);
- len_uchar = uchar_length(icu_converter, buff, nbytes);
+ len_uchar = uchar_length(iculib, icu_converter, buff, nbytes);
*buff_uchar = palloc((len_uchar + 1) * sizeof(**buff_uchar));
- len_uchar = uchar_convert(icu_converter,
+ len_uchar = uchar_convert(iculib, icu_converter,
*buff_uchar, len_uchar + 1, buff, nbytes);
return len_uchar;
@@ -2766,30 +2840,32 @@ icu_to_uchar(UChar **buff_uchar, const char *buff, size_t nbytes)
* The result string is nul-terminated.
*/
int32_t
-icu_from_uchar(char **result, const UChar *buff_uchar, int32_t len_uchar)
+icu_from_uchar(pg_icu_library *iculib, char **result, const UChar *buff_uchar,
+ int32_t len_uchar)
{
UErrorCode status;
int32_t len_result;
- init_icu_converter();
+ init_icu_converter(iculib);
status = U_ZERO_ERROR;
- len_result = ucnv_fromUChars(icu_converter, NULL, 0,
- buff_uchar, len_uchar, &status);
+ len_result = iculib->fromUChars(icu_converter, NULL, 0,
+ buff_uchar, len_uchar, &status);
if (U_FAILURE(status) && status != U_BUFFER_OVERFLOW_ERROR)
ereport(ERROR,
(errmsg("%s failed: %s", "ucnv_fromUChars",
- u_errorName(status))));
+ iculib->errorName(status))));
*result = palloc(len_result + 1);
status = U_ZERO_ERROR;
- len_result = ucnv_fromUChars(icu_converter, *result, len_result + 1,
- buff_uchar, len_uchar, &status);
+ len_result = iculib->fromUChars(icu_converter, *result,
+ len_result + 1, buff_uchar,
+ len_uchar, &status);
if (U_FAILURE(status))
ereport(ERROR,
(errmsg("%s failed: %s", "ucnv_fromUChars",
- u_errorName(status))));
+ iculib->errorName(status))));
return len_result;
}
@@ -2805,7 +2881,8 @@ icu_from_uchar(char **result, const UChar *buff_uchar, int32_t len_uchar)
*/
pg_attribute_unused()
static void
-icu_set_collation_attributes(UCollator *collator, const char *loc)
+icu_set_collation_attributes(pg_icu_library *iculib, UCollator *collator,
+ const char *loc)
{
char *str = asc_tolower(loc, strlen(loc));
@@ -2879,7 +2956,7 @@ icu_set_collation_attributes(UCollator *collator, const char *loc)
status = U_ILLEGAL_ARGUMENT_ERROR;
if (status == U_ZERO_ERROR)
- ucol_setAttribute(collator, uattr, uvalue, &status);
+ iculib->setAttribute(collator, uattr, uvalue, &status);
/*
* Pretend the error came from ucol_open(), for consistent error
@@ -2888,7 +2965,7 @@ icu_set_collation_attributes(UCollator *collator, const char *loc)
if (U_FAILURE(status))
ereport(ERROR,
(errmsg("could not open collator for locale \"%s\": %s",
- loc, u_errorName(status))));
+ loc, iculib->errorName(status))));
}
}
}
diff --git a/src/include/utils/pg_locale_internal.h b/src/include/utils/pg_locale_internal.h
index 33465ad92d..54445d8b87 100644
--- a/src/include/utils/pg_locale_internal.h
+++ b/src/include/utils/pg_locale_internal.h
@@ -14,6 +14,8 @@
#define _PG_LOCALE_INTERNAL_
#ifdef USE_ICU
+#include <unicode/ubrk.h>
+#include <unicode/ucnv.h>
#include <unicode/ucol.h>
#endif
@@ -30,6 +32,104 @@
#endif
#endif
+#ifdef USE_ICU
+/*
+ * An ICU library version that we're either linked against or have loaded at
+ * runtime.
+ */
+typedef struct pg_icu_library
+{
+ int major_version;
+ int minor_version;
+ void (*getICUVersion) (UVersionInfo info);
+ void (*getUnicodeVersion) (UVersionInfo into);
+ void (*getCLDRVersion) (UVersionInfo info, UErrorCode *status);
+ UCollator *(*openCollator) (const char *loc, UErrorCode *status);
+ void (*closeCollator) (UCollator *coll);
+ void (*getCollatorVersion) (const UCollator *coll, UVersionInfo info);
+ void (*getUCAVersion) (const UCollator *coll, UVersionInfo info);
+ void (*versionToString) (const UVersionInfo versionArray,
+ char *versionString);
+ UCollationResult (*strcoll) (const UCollator *coll,
+ const UChar *source,
+ int32_t sourceLength,
+ const UChar *target,
+ int32_t targetLength);
+ UCollationResult (*strcollUTF8) (const UCollator *coll,
+ const char *source,
+ int32_t sourceLength,
+ const char *target,
+ int32_t targetLength,
+ UErrorCode *status);
+ int32_t (*getSortKey) (const UCollator *coll,
+ const UChar *source,
+ int32_t sourceLength,
+ uint8_t *result,
+ int32_t resultLength);
+ int32_t (*nextSortKeyPart) (const UCollator *coll,
+ UCharIterator *iter,
+ uint32_t state[2],
+ uint8_t *dest,
+ int32_t count,
+ UErrorCode *status);
+ void (*setUTF8) (UCharIterator *iter,
+ const char *s,
+ int32_t length);
+ const char *(*errorName) (UErrorCode code);
+ int32_t (*strToUpper) (UChar *dest,
+ int32_t destCapacity,
+ const UChar *src,
+ int32_t srcLength,
+ const char *locale,
+ UErrorCode *pErrorCode);
+ int32_t (*strToLower) (UChar *dest,
+ int32_t destCapacity,
+ const UChar *src,
+ int32_t srcLength,
+ const char *locale,
+ UErrorCode *pErrorCode);
+ int32_t (*strToTitle) (UChar *dest,
+ int32_t destCapacity,
+ const UChar *src,
+ int32_t srcLength,
+ UBreakIterator *titleIter,
+ const char *locale,
+ UErrorCode *pErrorCode);
+ void (*setAttribute) (UCollator *coll,
+ UColAttribute attr,
+ UColAttributeValue value,
+ UErrorCode *status);
+ UConverter *(*openConverter) (const char *converterName,
+ UErrorCode * err);
+ void (*closeConverter) (UConverter *converter);
+ int32_t (*fromUChars) (UConverter *cnv,
+ char *dest,
+ int32_t destCapacity,
+ const UChar *src,
+ int32_t srcLength,
+ UErrorCode *pErrorCode);
+ int32_t (*toUChars) (UConverter *cnv,
+ UChar *dest,
+ int32_t destCapacity,
+ const char *src,
+ int32_t srcLength,
+ UErrorCode *pErrorCode);
+ int32_t (*toLanguageTag) (const char *localeID,
+ char *langtag,
+ int32_t langtagCapacity,
+ UBool strict,
+ UErrorCode *err);
+ int32_t (*getDisplayName) (const char *localeID,
+ const char *inLocaleID,
+ UChar *result,
+ int32_t maxResultSize,
+ UErrorCode *err);
+ int32_t (*countAvailable) (void);
+ const char *(*getAvailable) (int32_t n);
+} pg_icu_library;
+
+#endif
+
/*
* We define our own wrapper around locale_t so we can keep the same
* function signatures for all builds, while not having to create a
@@ -53,7 +153,8 @@ struct pg_locale_struct
#ifdef USE_ICU
struct
{
- UCollator *ucol;
+ UCollator *ucol;
+ pg_icu_library *lib;
} icu;
#endif
int dummy; /* in case we have neither LOCALE_T nor ICU */
@@ -61,8 +162,14 @@ struct pg_locale_struct
};
#ifdef USE_ICU
-extern int32_t icu_to_uchar(UChar **buff_uchar, const char *buff, size_t nbytes);
-extern int32_t icu_from_uchar(char **result, const UChar *buff_uchar, int32_t len_uchar);
+#define PG_ICU_LIB(x) ((x)->info.icu.lib)
+#define PG_ICU_COL(x) ((x)->info.icu.ucol)
+
+extern pg_icu_library *get_builtin_icu_library(void);
+extern int32_t icu_to_uchar(pg_icu_library *lib, UChar **buff_uchar,
+ const char *buff, size_t nbytes);
+extern int32_t icu_from_uchar(pg_icu_library *lib, char **result,
+ const UChar *buff_uchar, int32_t len_uchar);
#endif
#endif /* _PG_LOCALE_INTERNAL_ */
--
2.34.1
[text/x-patch] v2-0005-Add-get_icu_library_hook.patch (6.2K, ../[email protected]/6-v2-0005-Add-get_icu_library_hook.patch)
download | inline diff:
From 7dca88aa9f51e7674151bb90506b5305f9107bac Mon Sep 17 00:00:00 2001
From: Jeff Davis <[email protected]>
Date: Mon, 5 Dec 2022 16:14:18 -0800
Subject: [PATCH v2 5/6] Add get_icu_library_hook.
Controls how ICU library symbols are loaded.
---
src/backend/commands/collationcmds.c | 6 +--
src/backend/utils/adt/pg_locale.c | 56 ++++++++++++++++++++++----
src/include/utils/pg_locale_internal.h | 10 ++++-
3 files changed, 61 insertions(+), 11 deletions(-)
diff --git a/src/backend/commands/collationcmds.c b/src/backend/commands/collationcmds.c
index 16b1bcbdc0..f439e832da 100644
--- a/src/backend/commands/collationcmds.c
+++ b/src/backend/commands/collationcmds.c
@@ -560,7 +560,7 @@ get_icu_language_tag(const char *localename)
{
char buf[ULOC_FULLNAME_CAPACITY];
UErrorCode status;
- pg_icu_library *iculib = get_builtin_icu_library();
+ pg_icu_library *iculib = get_icu_library(NULL, NULL, NULL);
status = U_ZERO_ERROR;
iculib->toLanguageTag(localename, buf, sizeof(buf), true, &status);
@@ -586,7 +586,7 @@ get_icu_locale_comment(const char *localename)
int32 len_uchar;
int32 i;
char *result;
- pg_icu_library *iculib = get_builtin_icu_library();
+ pg_icu_library *iculib = get_icu_library(NULL, NULL, NULL);
status = U_ZERO_ERROR;
len_uchar = iculib->getDisplayName(localename, "en",
@@ -811,7 +811,7 @@ pg_import_system_collations(PG_FUNCTION_ARGS)
#ifdef USE_ICU
{
int i;
- pg_icu_library *iculib = get_builtin_icu_library();
+ pg_icu_library *iculib = get_icu_library(NULL, NULL, NULL);
/*
* Start the loop at -1 to sneak in the root locale without too much
diff --git a/src/backend/utils/adt/pg_locale.c b/src/backend/utils/adt/pg_locale.c
index 4daff8b7b5..b75a825df6 100644
--- a/src/backend/utils/adt/pg_locale.c
+++ b/src/backend/utils/adt/pg_locale.c
@@ -109,6 +109,34 @@ char *localized_full_days[7 + 1];
char *localized_abbrev_months[12 + 1];
char *localized_full_months[12 + 1];
+/*
+ * get_icu_library_hook can be set to control how the pg_icu_library is
+ * constructed inside a pg_locale_t structure, and therefore which specific
+ * ICU symbols are called.
+ *
+ * Without the hook, Postgres constructs the pg_icu_library from the version
+ * of ICU that Postgres is linked against at build time.
+ *
+ * The hook can instead load the ICU symbols from a different version of the
+ * ICU library on the system, which can avoid problems when the collation
+ * subtly changes across different versions of ICU.
+ *
+ * If the hook returns true, it indicates that it has successfully filled in
+ * the pg_icu_library structure that was passed in. If it returns false,
+ * Postgres will fill in the structure itself. The version of the collation
+ * returned does not need to match exactly the version that was passed in;
+ * though if not, Postgres will issue a WARNING.
+ *
+ * XXX: For now, the only information the hook has access to is the ICU
+ * collation name, ICU ctype, and the ICU version string that was obtained at
+ * the time the collation was created (or when it was last refreshed). We
+ * should consider what other information can be provided to allow for greater
+ * control which library is loaded.
+ */
+#ifdef USE_ICU
+get_icu_library_hook_type get_icu_library_hook = NULL;
+#endif
+
/* indicates whether locale information cache is valid */
static bool CurrentLocaleConvValid = false;
static bool CurrentLCTimeValid = false;
@@ -1463,18 +1491,15 @@ report_newlocale_failure(const char *localename)
#endif /* HAVE_LOCALE_T */
#ifdef USE_ICU
-pg_icu_library *
-get_builtin_icu_library()
+static bool
+get_builtin_icu_library(pg_icu_library *lib)
{
- pg_icu_library *lib;
-
/*
* These assignments will fail to compile if an incompatible API change is
* made to some future version of ICU, at which point we might need to
* consider special treatment for different major version ranges, with
* intermediate trampoline functions.
*/
- lib = palloc0(sizeof(*lib));
lib->getICUVersion = u_getVersion;
lib->getUnicodeVersion = u_getUnicodeVersion;
lib->getCLDRVersion = ulocdata_getCLDRVersion;
@@ -1512,8 +1537,25 @@ get_builtin_icu_library()
StaticAssertStmt(U_MAX_VERSION_LENGTH == 4,
"ucol_getVersion output buffer size changed incompatibly");
+ return true;
+}
+
+pg_icu_library *
+get_icu_library(const char *collate, const char *ctype, const char *version)
+{
+ pg_icu_library *lib = palloc0(sizeof(*lib));
+ bool filled = false;
+
+ if (get_icu_library_hook != NULL)
+ filled = get_icu_library_hook(lib, collate, ctype, version);
+
+ if(!filled)
+ filled = get_builtin_icu_library(lib);
+
+ Assert(filled);
return lib;
}
+
#endif
/*
@@ -1607,7 +1649,7 @@ pg_newlocale(char provider, bool isdefault, bool deterministic,
{
UCollator *collator;
UErrorCode status;
- pg_icu_library *iculib = get_builtin_icu_library();
+ pg_icu_library *iculib = get_icu_library(collate, ctype, version);
/* collator may be leaked if we encounter an error */
@@ -1926,7 +1968,7 @@ get_collation_actual_version(char collprovider, const char *collcollate)
UErrorCode status;
UVersionInfo versioninfo;
char buf[U_MAX_VERSION_STRING_LENGTH];
- pg_icu_library *iculib = get_builtin_icu_library();
+ pg_icu_library *iculib = get_icu_library(NULL, NULL, NULL);
status = U_ZERO_ERROR;
collator = iculib->openCollator(collcollate, &status);
diff --git a/src/include/utils/pg_locale_internal.h b/src/include/utils/pg_locale_internal.h
index 54445d8b87..1be0205391 100644
--- a/src/include/utils/pg_locale_internal.h
+++ b/src/include/utils/pg_locale_internal.h
@@ -162,10 +162,18 @@ struct pg_locale_struct
};
#ifdef USE_ICU
+
+typedef bool (*get_icu_library_hook_type)(
+ pg_icu_library *lib, const char *collate, const char *ctype,
+ const char *version);
+
+extern PGDLLIMPORT get_icu_library_hook_type get_icu_library_hook;
+
#define PG_ICU_LIB(x) ((x)->info.icu.lib)
#define PG_ICU_COL(x) ((x)->info.icu.ucol)
-extern pg_icu_library *get_builtin_icu_library(void);
+extern pg_icu_library *get_icu_library(const char *collate, const char *ctype,
+ const char *version);
extern int32_t icu_to_uchar(pg_icu_library *lib, UChar **buff_uchar,
const char *buff, size_t nbytes);
extern int32_t icu_from_uchar(pg_icu_library *lib, char **result,
--
2.34.1
view thread (137+ messages) latest in thread
reply
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: [email protected]
Cc: [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected]
Subject: Re: Collation version tracking for macOS
In-Reply-To: <[email protected]>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox