public inbox for [email protected]
help / color / mirror / Atom feedRe: Built-in CTYPE provider
9+ messages / 4 participants
[nested] [flat]
* Re: Built-in CTYPE provider
@ 2023-12-29 02:57 Jeff Davis <[email protected]>
2024-01-09 01:17 ` Re: Built-in CTYPE provider Jeremy Schneider <[email protected]>
2024-01-09 22:17 ` Re: Built-in CTYPE provider Jeremy Schneider <[email protected]>
2024-01-10 22:56 ` Re: Built-in CTYPE provider Daniel Verite <[email protected]>
0 siblings, 3 replies; 9+ messages in thread
From: Jeff Davis @ 2023-12-29 02:57 UTC (permalink / raw)
To: Daniel Verite <[email protected]>; +Cc: Robert Haas <[email protected]>; Jeremy Schneider <[email protected]>; pgsql-hackers
On Wed, 2023-12-27 at 17:26 -0800, Jeff Davis wrote:
> Attached is an implementation of a built-in provider for the "C.UTF-
> 8"
Attached a more complete version that fixes a few bugs, stabilizes the
tests, and improves the documentation. I optimized the performance, too
-- now it's beating both libc's "C.utf8" and ICU "en-US-x-icu" for both
collation and case mapping (numbers below).
It's really nice to finally be able to have platform-independent tests
that work on any UTF-8 database.
Simple character classification:
SELECT 'Á' ~ '[[:alpha:]]' COLLATE C_UTF8;
Case mapping is more interesting (note that accented characters are
being properly mapped, and it's using the titlecase variant "Dž"):
SELECT initcap('axxE áxxÉ DŽxxDŽ Džxxx džxxx' COLLATE C_UTF8);
initcap
--------------------------
Axxe Áxxé Džxxdž Džxxx Džxxx
Even more interesting -- test that non-latin characters can still be a
member of a case-insensitive range:
-- capital delta is member of lowercase range gamma to lambda
SELECT 'Δ' ~* '[γ-λ]' COLLATE C_UTF8;
-- small delta is member of uppercase range gamma to lambda
SELECT 'δ' ~* '[Γ-Λ]' COLLATE C_UTF8;
Moreover, a lot of this behavior is locked in by strong Unicode
guarantees like [1] and [2]. Behavior that can change probably won't
change very often, and in any case will be tied to a PG major version.
All of these behaviors are very close to what glibc "C.utf8" does on my
machine. The case transformations are identical (except titlecasing
because libc doesn't support it). The character classifications have
some differences, which might be worth discussing, but I didn't see
anything terribly concerning (I am following the unicode
recommendations[3] on this topic).
Performance:
Sotring 10M strings:
libc "C" 14s
builtin C_UTF8 14s
libc "C.utf8" 20s
ICU "en-US-x-icu" 31s
Running UPPER() on 10M strings:
libc "C" 03s
builtin C_UTF8 07s
libc "C.utf8" 08s
ICU "en-US-x-icu" 15s
I didn't investigate or optimize regexes / pattern matching yet, but I
can do similar optimizations if there's any gap.
Note that I implemented the "simple" case mapping (which is what glibc
does) and the "posix compatible"[3] flavor of character classification
(which is closer to what glibc does than the "standard" flavor"). I
opted to use title case mapping for initcap(), which is a difference
from libc and I may go back to just upper/lower. These seem like
reasonable choices if we're going to name the locale after C.UTF-8.
Regards,
Jeff Davis
[1] https://www.unicode.org/policies/stability_policy.html#Case_Pair
[2] https://www.unicode.org/policies/stability_policy.html#Identity
[3] http://www.unicode.org/reports/tr18/#Compatibility_Properties
Attachments:
[text/x-patch] v15-0001-Minor-cleanup-for-unicode-update-build-and-test.patch (7.4K, ../../[email protected]/2-v15-0001-Minor-cleanup-for-unicode-update-build-and-test.patch)
download | inline diff:
From dc433f10296abdb6837eceb93f096ed23c7115f5 Mon Sep 17 00:00:00 2001
From: Jeff Davis <[email protected]>
Date: Wed, 22 Nov 2023 10:38:46 -0800
Subject: [PATCH v15 1/5] Minor cleanup for unicode-update build and test.
---
src/common/unicode/Makefile | 6 ++--
src/common/unicode/category_test.c | 18 ++++++------
src/common/unicode/meson.build | 44 +++++++++++++++---------------
3 files changed, 34 insertions(+), 34 deletions(-)
diff --git a/src/common/unicode/Makefile b/src/common/unicode/Makefile
index 30cd75cc6a..04d81dd5cb 100644
--- a/src/common/unicode/Makefile
+++ b/src/common/unicode/Makefile
@@ -21,7 +21,7 @@ CPPFLAGS += $(ICU_CFLAGS)
# By default, do nothing.
all:
-update-unicode: unicode_category_table.h unicode_norm_table.h unicode_nonspacing_table.h unicode_east_asian_fw_table.h unicode_normprops_table.h unicode_norm_hashfunc.h unicode_version.h
+update-unicode: unicode_category_table.h unicode_east_asian_fw_table.h unicode_nonspacing_table.h unicode_norm_hashfunc.h unicode_norm_table.h unicode_normprops_table.h unicode_version.h
mv $^ $(top_srcdir)/src/include/common/
$(MAKE) category-check
$(MAKE) normalization-check
@@ -29,7 +29,7 @@ update-unicode: unicode_category_table.h unicode_norm_table.h unicode_nonspacing
# These files are part of the Unicode Character Database. Download
# them on demand. The dependency on Makefile.global is for
# UNICODE_VERSION.
-UnicodeData.txt EastAsianWidth.txt DerivedNormalizationProps.txt CompositionExclusions.txt NormalizationTest.txt: $(top_builddir)/src/Makefile.global
+CompositionExclusions.txt DerivedNormalizationProps.txt EastAsianWidth.txt NormalizationTest.txt UnicodeData.txt: $(top_builddir)/src/Makefile.global
$(DOWNLOAD) https://www.unicode.org/Public/$(UNICODE_VERSION)/ucd/$(@F)
unicode_version.h: generate-unicode_version.pl
@@ -82,4 +82,4 @@ clean:
rm -f $(OBJS) category_test category_test.o norm_test norm_test.o
distclean: clean
- rm -f UnicodeData.txt EastAsianWidth.txt CompositionExclusions.txt NormalizationTest.txt norm_test_table.h unicode_norm_table.h
+ rm -f CompositionExclusions.txt DerivedNormalizationProps.txt EastAsianWidth.txt NormalizationTest.txt UnicodeData.txt norm_test_table.h unicode_category_table.h unicode_norm_table.h
diff --git a/src/common/unicode/category_test.c b/src/common/unicode/category_test.c
index ba62716d45..d9ea806eb8 100644
--- a/src/common/unicode/category_test.c
+++ b/src/common/unicode/category_test.c
@@ -54,8 +54,8 @@ main(int argc, char **argv)
int pg_skipped_codepoints = 0;
int icu_skipped_codepoints = 0;
- printf("Postgres Unicode Version:\t%s\n", PG_UNICODE_VERSION);
- printf("ICU Unicode Version:\t\t%s\n", U_UNICODE_VERSION);
+ printf("category_test: Postgres Unicode version:\t%s\n", PG_UNICODE_VERSION);
+ printf("category_test: ICU Unicode version:\t\t%s\n", U_UNICODE_VERSION);
for (UChar32 code = 0; code <= 0x10ffff; code++)
{
@@ -79,11 +79,11 @@ main(int argc, char **argv)
icu_skipped_codepoints++;
else
{
- printf("FAILURE for codepoint %06x\n", code);
- printf("Postgres category: %02d %s %s\n", pg_category,
+ printf("category_test: FAILURE for codepoint 0x%06x\n", code);
+ printf("category_test: Postgres category: %02d %s %s\n", pg_category,
unicode_category_abbrev(pg_category),
unicode_category_string(pg_category));
- printf("ICU category: %02d %s %s\n", icu_category,
+ printf("category_test: ICU category: %02d %s %s\n", icu_category,
unicode_category_abbrev(icu_category),
unicode_category_string(icu_category));
printf("\n");
@@ -93,16 +93,16 @@ main(int argc, char **argv)
}
if (pg_skipped_codepoints > 0)
- printf("Skipped %d codepoints unassigned in Postgres due to Unicode version mismatch.\n",
+ printf("category_test: skipped %d codepoints unassigned in Postgres due to Unicode version mismatch\n",
pg_skipped_codepoints);
if (icu_skipped_codepoints > 0)
- printf("Skipped %d codepoints unassigned in ICU due to Unicode version mismatch.\n",
+ printf("category_test: skipped %d codepoints unassigned in ICU due to Unicode version mismatch\n",
icu_skipped_codepoints);
- printf("category_test: All tests successful!\n");
+ printf("category_test: success\n");
exit(0);
#else
- printf("ICU support required for test; skipping.\n");
+ printf("category_test: ICU support required for test; skipping\n");
exit(0);
#endif
}
diff --git a/src/common/unicode/meson.build b/src/common/unicode/meson.build
index 6af46122c4..e8cfdc1df4 100644
--- a/src/common/unicode/meson.build
+++ b/src/common/unicode/meson.build
@@ -11,7 +11,7 @@ endif
# These files are part of the Unicode Character Database. Download them on
# demand.
-foreach f : ['UnicodeData.txt', 'EastAsianWidth.txt', 'DerivedNormalizationProps.txt', 'CompositionExclusions.txt', 'NormalizationTest.txt']
+foreach f : ['CompositionExclusions.txt', 'DerivedNormalizationProps.txt', 'EastAsianWidth.txt', 'NormalizationTest.txt', 'UnicodeData.txt']
url = unicode_baseurl.format(UNICODE_VERSION, f)
target = custom_target(f,
output: f,
@@ -24,15 +24,6 @@ endforeach
update_unicode_targets = []
-update_unicode_targets += \
- custom_target('unicode_version.h',
- output: ['unicode_version.h'],
- command: [
- perl, files('generate-unicode_version.pl'),
- '--outdir', '@OUTDIR@', '--version', UNICODE_VERSION],
- build_by_default: false,
- )
-
update_unicode_targets += \
custom_target('unicode_category_table.h',
input: [unicode_data['UnicodeData.txt']],
@@ -44,14 +35,12 @@ update_unicode_targets += \
)
update_unicode_targets += \
- custom_target('unicode_norm_table.h',
- input: [unicode_data['UnicodeData.txt'], unicode_data['CompositionExclusions.txt']],
- output: ['unicode_norm_table.h', 'unicode_norm_hashfunc.h'],
- depend_files: perfect_hash_pm,
- command: [
- perl, files('generate-unicode_norm_table.pl'),
- '--outdir', '@OUTDIR@', '@INPUT@'],
+ custom_target('unicode_east_asian_fw_table.h',
+ input: [unicode_data['EastAsianWidth.txt']],
+ output: ['unicode_east_asian_fw_table.h'],
+ command: [perl, files('generate-unicode_east_asian_fw_table.pl'), '@INPUT@'],
build_by_default: false,
+ capture: true,
)
update_unicode_targets += \
@@ -65,12 +54,14 @@ update_unicode_targets += \
)
update_unicode_targets += \
- custom_target('unicode_east_asian_fw_table.h',
- input: [unicode_data['EastAsianWidth.txt']],
- output: ['unicode_east_asian_fw_table.h'],
- command: [perl, files('generate-unicode_east_asian_fw_table.pl'), '@INPUT@'],
+ custom_target('unicode_norm_table.h',
+ input: [unicode_data['UnicodeData.txt'], unicode_data['CompositionExclusions.txt']],
+ output: ['unicode_norm_table.h', 'unicode_norm_hashfunc.h'],
+ depend_files: perfect_hash_pm,
+ command: [
+ perl, files('generate-unicode_norm_table.pl'),
+ '--outdir', '@OUTDIR@', '@INPUT@'],
build_by_default: false,
- capture: true,
)
update_unicode_targets += \
@@ -83,6 +74,15 @@ update_unicode_targets += \
capture: true,
)
+update_unicode_targets += \
+ custom_target('unicode_version.h',
+ output: ['unicode_version.h'],
+ command: [
+ perl, files('generate-unicode_version.pl'),
+ '--outdir', '@OUTDIR@', '--version', UNICODE_VERSION],
+ build_by_default: false,
+ )
+
norm_test_table = custom_target('norm_test_table.h',
input: [unicode_data['NormalizationTest.txt']],
output: ['norm_test_table.h'],
--
2.34.1
[text/x-patch] v15-0002-Add-Unicode-property-tables.patch (91.5K, ../../[email protected]/3-v15-0002-Add-Unicode-property-tables.patch)
download | inline diff:
From 10ab7ca76bfe7dff10e903bc61c509a781e4b74b Mon Sep 17 00:00:00 2001
From: Jeff Davis <[email protected]>
Date: Sat, 18 Nov 2023 15:34:24 -0800
Subject: [PATCH v15 2/5] Add Unicode property tables.
---
src/common/unicode/Makefile | 6 +-
src/common/unicode/category_test.c | 302 +-
.../generate-unicode_category_table.pl | 203 +-
src/common/unicode/meson.build | 4 +-
src/common/unicode_category.c | 258 +-
src/include/common/unicode_category.h | 25 +-
src/include/common/unicode_category_table.h | 2532 +++++++++++++++++
7 files changed, 3264 insertions(+), 66 deletions(-)
diff --git a/src/common/unicode/Makefile b/src/common/unicode/Makefile
index 04d81dd5cb..27f0408d8b 100644
--- a/src/common/unicode/Makefile
+++ b/src/common/unicode/Makefile
@@ -29,13 +29,13 @@ update-unicode: unicode_category_table.h unicode_east_asian_fw_table.h unicode_n
# These files are part of the Unicode Character Database. Download
# them on demand. The dependency on Makefile.global is for
# UNICODE_VERSION.
-CompositionExclusions.txt DerivedNormalizationProps.txt EastAsianWidth.txt NormalizationTest.txt UnicodeData.txt: $(top_builddir)/src/Makefile.global
+CompositionExclusions.txt DerivedCoreProperties.txt DerivedNormalizationProps.txt EastAsianWidth.txt NormalizationTest.txt PropList.txt UnicodeData.txt: $(top_builddir)/src/Makefile.global
$(DOWNLOAD) https://www.unicode.org/Public/$(UNICODE_VERSION)/ucd/$(@F)
unicode_version.h: generate-unicode_version.pl
$(PERL) $< --version $(UNICODE_VERSION)
-unicode_category_table.h: generate-unicode_category_table.pl UnicodeData.txt
+unicode_category_table.h: generate-unicode_category_table.pl DerivedCoreProperties.txt PropList.txt UnicodeData.txt
$(PERL) $<
# Generation of conversion tables used for string normalization with
@@ -82,4 +82,4 @@ clean:
rm -f $(OBJS) category_test category_test.o norm_test norm_test.o
distclean: clean
- rm -f CompositionExclusions.txt DerivedNormalizationProps.txt EastAsianWidth.txt NormalizationTest.txt UnicodeData.txt norm_test_table.h unicode_category_table.h unicode_norm_table.h
+ rm -f CompositionExclusions.txt DerivedCoreProperties.txt DerivedNormalizationProps.txt EastAsianWidth.txt NormalizationTest.txt PropList.txt UnicodeData.txt norm_test_table.h unicode_category_table.h unicode_norm_table.h
diff --git a/src/common/unicode/category_test.c b/src/common/unicode/category_test.c
index d9ea806eb8..6451988e94 100644
--- a/src/common/unicode/category_test.c
+++ b/src/common/unicode/category_test.c
@@ -1,6 +1,7 @@
/*-------------------------------------------------------------------------
* category_test.c
- * Program to test Unicode general category functions.
+ * Program to test Unicode general category and character class
+ * functions.
*
* Portions Copyright (c) 2017-2023, PostgreSQL Global Development Group
*
@@ -14,23 +15,30 @@
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
-
#ifdef USE_ICU
#include <unicode/uchar.h>
#endif
-#include "common/unicode_category.h"
+#include <wctype.h>
+
#include "common/unicode_version.h"
+#include "common/unicode_category.h"
+
+#define LIBC_LOCALE "C.UTF-8"
+
+static int pg_unicode_version = 0;
+#ifdef USE_ICU
+static int icu_unicode_version = 0;
+#endif
/*
* Parse version into integer for easy comparison.
*/
-#ifdef USE_ICU
static int
parse_unicode_version(const char *version)
{
- int n,
- major,
- minor;
+ int major;
+ int minor;
+ int PG_USED_FOR_ASSERTS_ONLY n;
n = sscanf(version, "%d.%d", &major, &minor);
@@ -39,56 +47,160 @@ parse_unicode_version(const char *version)
return major * 100 + minor;
}
-#endif
+#ifdef USE_ICU
/*
- * Exhaustively test that the Unicode category for each codepoint matches that
- * returned by ICU.
+ * Test Postgres Unicode tables by comparing with ICU. Test the General
+ * Category, as well as the properties Alphabetic, Lowercase, Uppercase,
+ * White_Space, and Hex_Digit.
*/
-int
-main(int argc, char **argv)
+static void
+icu_test()
{
-#ifdef USE_ICU
- int pg_unicode_version = parse_unicode_version(PG_UNICODE_VERSION);
- int icu_unicode_version = parse_unicode_version(U_UNICODE_VERSION);
int pg_skipped_codepoints = 0;
int icu_skipped_codepoints = 0;
- printf("category_test: Postgres Unicode version:\t%s\n", PG_UNICODE_VERSION);
- printf("category_test: ICU Unicode version:\t\t%s\n", U_UNICODE_VERSION);
-
- for (UChar32 code = 0; code <= 0x10ffff; code++)
+ for (pg_wchar code = 0; code <= 0x10ffff; code++)
{
uint8_t pg_category = unicode_category(code);
uint8_t icu_category = u_charType(code);
+ /* Property tests */
+ bool prop_alphabetic = pg_u_prop_alphabetic(code);
+ bool prop_lowercase = pg_u_prop_lowercase(code);
+ bool prop_uppercase = pg_u_prop_uppercase(code);
+ bool prop_white_space = pg_u_prop_white_space(code);
+ bool prop_hex_digit = pg_u_prop_hex_digit(code);
+ bool prop_join_control = pg_u_prop_join_control(code);
+
+ bool icu_prop_alphabetic = u_hasBinaryProperty(
+ code, UCHAR_ALPHABETIC);
+ bool icu_prop_lowercase = u_hasBinaryProperty(
+ code, UCHAR_LOWERCASE);
+ bool icu_prop_uppercase = u_hasBinaryProperty(
+ code, UCHAR_UPPERCASE);
+ bool icu_prop_white_space = u_hasBinaryProperty(
+ code, UCHAR_WHITE_SPACE);
+ bool icu_prop_hex_digit = u_hasBinaryProperty(
+ code, UCHAR_HEX_DIGIT);
+ bool icu_prop_join_control = u_hasBinaryProperty(
+ code, UCHAR_JOIN_CONTROL);
+
+ /*
+ * Compare with ICU for TR #18 character classes using:
+ *
+ * https://unicode-org.github.io/icu-docs/apidoc/dev/icu4c/uchar_8h.html#details
+ *
+ * which describes how to use ICU to test for membership in regex
+ * character classes ("Standard", not "POSIX Compatible").
+ *
+ * NB: the document suggests testing for some properties such as
+ * UCHAR_POSIX_ALNUM, but that doesn't mean that we're testing for the
+ * "POSIX Compatible" character classes.
+ */
+ bool isalpha = pg_u_isalpha(code);
+ bool islower = pg_u_islower(code);
+ bool isupper = pg_u_isupper(code);
+ bool ispunct = pg_u_ispunct(code, false);
+ bool isdigit = pg_u_isdigit(code, false);
+ bool isxdigit = pg_u_isxdigit(code, false);
+ bool isalnum = pg_u_isalnum(code, false);
+ bool isspace = pg_u_isspace(code);
+ bool isblank = pg_u_isblank(code);
+ bool iscntrl = pg_u_iscntrl(code);
+ bool isgraph = pg_u_isgraph(code);
+ bool isprint = pg_u_isprint(code);
+
+ bool icu_isalpha = u_isUAlphabetic(code);
+ bool icu_islower = u_isULowercase(code);
+ bool icu_isupper = u_isUUppercase(code);
+ bool icu_ispunct = u_ispunct(code);
+ bool icu_isdigit = u_isdigit(code);
+ bool icu_isxdigit = u_hasBinaryProperty(code,
+ UCHAR_POSIX_XDIGIT);
+ bool icu_isalnum = u_hasBinaryProperty(code,
+ UCHAR_POSIX_ALNUM);
+ bool icu_isspace = u_isUWhiteSpace(code);
+ bool icu_isblank = u_isblank(code);
+ bool icu_iscntrl = icu_category == PG_U_CONTROL;
+ bool icu_isgraph = u_hasBinaryProperty(code,
+ UCHAR_POSIX_GRAPH);
+ bool icu_isprint = u_hasBinaryProperty(code,
+ UCHAR_POSIX_PRINT);
+
+ /*
+ * A version mismatch means that some assigned codepoints in the newer
+ * version may be unassigned in the older version. That's OK, though
+ * the test will not cover those codepoints marked unassigned in the
+ * older version (that is, it will no longer be an exhaustive test).
+ */
+ if (pg_category == PG_U_UNASSIGNED &&
+ icu_category != PG_U_UNASSIGNED &&
+ pg_unicode_version < icu_unicode_version)
+ {
+ pg_skipped_codepoints++;
+ continue;
+ }
+
+ if (icu_category == PG_U_UNASSIGNED &&
+ pg_category != PG_U_UNASSIGNED &&
+ icu_unicode_version < pg_unicode_version)
+ {
+ icu_skipped_codepoints++;
+ continue;
+ }
+
if (pg_category != icu_category)
{
- /*
- * A version mismatch means that some assigned codepoints in the
- * newer version may be unassigned in the older version. That's
- * OK, though the test will not cover those codepoints marked
- * unassigned in the older version (that is, it will no longer be
- * an exhaustive test).
- */
- if (pg_category == PG_U_UNASSIGNED &&
- pg_unicode_version < icu_unicode_version)
- pg_skipped_codepoints++;
- else if (icu_category == PG_U_UNASSIGNED &&
- icu_unicode_version < pg_unicode_version)
- icu_skipped_codepoints++;
- else
- {
- printf("category_test: FAILURE for codepoint 0x%06x\n", code);
- printf("category_test: Postgres category: %02d %s %s\n", pg_category,
- unicode_category_abbrev(pg_category),
- unicode_category_string(pg_category));
- printf("category_test: ICU category: %02d %s %s\n", icu_category,
- unicode_category_abbrev(icu_category),
- unicode_category_string(icu_category));
- printf("\n");
- exit(1);
- }
+ printf("category_test: FAILURE for codepoint 0x%06x\n", code);
+ printf("category_test: Postgres category: %02d %s %s\n", pg_category,
+ unicode_category_abbrev(pg_category),
+ unicode_category_string(pg_category));
+ printf("category_test: ICU category: %02d %s %s\n", icu_category,
+ unicode_category_abbrev(icu_category),
+ unicode_category_string(icu_category));
+ printf("\n");
+ exit(1);
+ }
+
+ if (prop_alphabetic != icu_prop_alphabetic ||
+ prop_lowercase != icu_prop_lowercase ||
+ prop_uppercase != icu_prop_uppercase ||
+ prop_white_space != icu_prop_white_space ||
+ prop_hex_digit != icu_prop_hex_digit ||
+ prop_join_control != icu_prop_join_control)
+ {
+ printf("category_test: FAILURE for codepoint 0x%06x\n", code);
+ printf("category_test: Postgres property alphabetic/lowercase/uppercase/white_space/hex_digit/join_control: %d/%d/%d/%d/%d/%d\n",
+ prop_alphabetic, prop_lowercase, prop_uppercase,
+ prop_white_space, prop_hex_digit, prop_join_control);
+ printf("category_test: ICU property alphabetic/lowercase/uppercase/white_space/hex_digit/join_control: %d/%d/%d/%d/%d/%d\n",
+ icu_prop_alphabetic, icu_prop_lowercase, icu_prop_uppercase,
+ icu_prop_white_space, icu_prop_hex_digit, icu_prop_join_control);
+ printf("\n");
+ exit(1);
+ }
+
+ if (isalpha != icu_isalpha ||
+ islower != icu_islower ||
+ isupper != icu_isupper ||
+ ispunct != icu_ispunct ||
+ isdigit != icu_isdigit ||
+ isxdigit != icu_isxdigit ||
+ isalnum != icu_isalnum ||
+ isspace != icu_isspace ||
+ isblank != icu_isblank ||
+ iscntrl != icu_iscntrl ||
+ isgraph != icu_isgraph ||
+ isprint != icu_isprint)
+ {
+ printf("category_test: FAILURE for codepoint 0x%06x\n", code);
+ printf("category_test: Postgres class alpha/lower/upper/punct/digit/xdigit/alnum/space/blank/cntrl/graph/print: %d/%d/%d/%d/%d/%d/%d/%d/%d/%d/%d/%d\n",
+ isalpha, islower, isupper, ispunct, isdigit, isxdigit, isalnum, isspace, isblank, iscntrl, isgraph, isprint);
+ printf("category_test: ICU class alpha/lower/upper/punct/digit/xdigit/alnum/space/blank/cntrl/graph/print: %d/%d/%d/%d/%d/%d/%d/%d/%d/%d/%d/%d\n",
+ icu_isalpha, icu_islower, icu_isupper, icu_ispunct, icu_isdigit, icu_isxdigit, icu_isalnum, icu_isspace, icu_isblank, icu_iscntrl, icu_isgraph, icu_isprint);
+ printf("\n");
+ exit(1);
}
}
@@ -99,10 +211,104 @@ main(int argc, char **argv)
printf("category_test: skipped %d codepoints unassigned in ICU due to Unicode version mismatch\n",
icu_skipped_codepoints);
- printf("category_test: success\n");
- exit(0);
+ printf("category_test: ICU test successful\n");
+}
+#endif
+
+/*
+ * For libc, test only some characters for membership in the punctuation
+ * class. We have no guarantee that all characters will obey the same rules as
+ * pg_u_ispunct_posix(), though some coverage is still useful.
+ */
+static const unsigned char test_punct[] = {
+ ',', '$', '"', 0x85, 0x00, 'b', '&', 'Z', ' ', '\t', '\n'
+};
+
+/*
+ * Test what we can for libc, which is limited but still useful to cover the
+ * _posix-variant functions.
+ */
+static void
+libc_test()
+{
+ char * libc_locale = setlocale(LC_CTYPE, LIBC_LOCALE);
+
+ if (!libc_locale)
+ {
+ printf("category_test: libc locale \"%s\" not available; skipping\n", LIBC_LOCALE);
+ return;
+ }
+
+ /* non-exhaustive test of pg_u_ispunct_posix() */
+ for (int i = 0; i < sizeof(test_punct)/sizeof(test_punct[0]); i++)
+ {
+ pg_wchar code = (pg_wchar) test_punct[i];
+ bool ispunct = pg_u_ispunct(code, true);
+ bool libc_ispunct = iswpunct(code);
+
+ if (ispunct != libc_ispunct)
+ {
+ printf("category_test: FAILURE for codepoint 0x%06x\n", code);
+ printf("category_test: Postgres ispunct_posix: %d\n", ispunct);
+ printf("category_test: libc iswpunct: %d\n", libc_ispunct);
+ printf("\n");
+ exit(1);
+ }
+ }
+
+ for (pg_wchar code = 0; code <= 0x10ffff; code++)
+ {
+ uint8_t pg_category = unicode_category(code);
+
+ bool isalpha = pg_u_isalpha(code);
+ bool isdigit = pg_u_isdigit(code, true);
+ bool isxdigit = pg_u_isxdigit(code, true);
+ bool isalnum = pg_u_isalnum(code, true);
+
+ bool libc_isdigit = iswdigit(code);
+ bool libc_isxdigit = iswxdigit(code);
+
+ if (pg_category == PG_U_UNASSIGNED)
+ continue;
+
+ /* check that alnum is the same as isdigit OR isalpha */
+ if (((isdigit || isalpha) && !isalnum) ||
+ (!(isdigit || isalpha) && isalnum))
+ {
+ printf("category_test: FAILURE for codepoint 0x%06x\n", code);
+ printf("category_test: isalnum inconsistent: isalpha/isdigit/isalnum: %d/%d/%d\n",
+ isalpha, isdigit, isalnum);
+ exit(1);
+ }
+
+ if (isdigit != libc_isdigit ||
+ isxdigit != libc_isxdigit)
+ {
+ printf("category_test: FAILURE for codepoint 0x%06x\n", code);
+ printf("category_test: Postgres class digit/xdigit: %d/%d\n",
+ isdigit, isxdigit);
+ printf("category_test: libc class digit/xdigit: %d/%d\n",
+ libc_isdigit, libc_isxdigit);
+ printf("\n");
+ exit(1);
+ }
+ }
+}
+
+int
+main(int argc, char **argv)
+{
+ pg_unicode_version = parse_unicode_version(PG_UNICODE_VERSION);
+ printf("category_test: Postgres Unicode version:\t%s\n", PG_UNICODE_VERSION);
+
+ libc_test();
+
+#ifdef USE_ICU
+ icu_unicode_version = parse_unicode_version(U_UNICODE_VERSION);
+ printf("category_test: ICU Unicode version:\t\t%s\n", U_UNICODE_VERSION);
+
+ icu_test();
#else
- printf("category_test: ICU support required for test; skipping\n");
- exit(0);
+ printf("category_test: ICU not available; skipping\n");
#endif
}
diff --git a/src/common/unicode/generate-unicode_category_table.pl b/src/common/unicode/generate-unicode_category_table.pl
index 992b877ede..9545728443 100644
--- a/src/common/unicode/generate-unicode_category_table.pl
+++ b/src/common/unicode/generate-unicode_category_table.pl
@@ -120,8 +120,6 @@ if ($range_category ne $CATEGORY_UNASSIGNED) {
category => $range_category});
}
-my $num_ranges = scalar @category_ranges;
-
# See: https://www.unicode.org/reports/tr44/#General_Category_Values
my $categories = {
Cn => 'PG_U_UNASSIGNED',
@@ -156,11 +154,98 @@ my $categories = {
Pf => 'PG_U_FINAL_PUNCTUATION'
};
-# Start writing out the output files
+# Find White_Space and Hex_Digit characters
+my @white_space = ();
+my @hex_digits = ();
+my @join_control = ();
+open($FH, '<', "$output_path/PropList.txt")
+ or die "Could not open $output_path/PropList.txt: $!.";
+while (my $line = <$FH>)
+{
+ my $pattern = qr/([0-9A-F\.]+)\s*;\s*(\w+)\s*#.*/s;
+ next unless $line =~ $pattern;
+
+ my $code = $line =~ s/$pattern/$1/rg;
+ my $property = $line =~ s/$pattern/$2/rg;
+ my $start;
+ my $end;
+
+ if ($code =~ /\.\./) {
+ # code range
+ my @sp = split /\.\./, $code;
+ $start = hex($sp[0]);
+ $end = hex($sp[1]);
+ } else {
+ # single code point
+ $start = hex($code);
+ $end = hex($code);
+ }
+
+ if ($property eq "White_Space") {
+ push @white_space, {start => $start, end => $end};
+ }
+ elsif ($property eq "Hex_Digit") {
+ push @hex_digits, {start => $start, end => $end};
+ }
+ elsif ($property eq "Join_Control") {
+ push @join_control, {start => $start, end => $end};
+ }
+}
+
+# Find Alphabetic, Lowercase, and Uppercase characters
+my @alphabetic = ();
+my @lowercase = ();
+my @uppercase = ();
+open($FH, '<', "$output_path/DerivedCoreProperties.txt")
+ or die "Could not open $output_path/DerivedCoreProperties.txt: $!.";
+while (my $line = <$FH>)
+{
+ my $pattern = qr/^([0-9A-F\.]+)\s*;\s*(\w+)\s*#.*$/s;
+ next unless $line =~ $pattern;
+
+ my $code = $line =~ s/$pattern/$1/rg;
+ my $property = $line =~ s/$pattern/$2/rg;
+ my $start;
+ my $end;
+
+ if ($code =~ /\.\./) {
+ # code range
+ my @sp = split /\.\./, $code;
+ die "line: {$line} code: {$code} sp[0] {$sp[0]} sp[1] {$sp[1]}"
+ unless $sp[0] =~ /^[0-9A-F]+$/ && $sp[1] =~ /^[0-9A-F]+$/;
+ $start = hex($sp[0]);
+ $end = hex($sp[1]);
+ } else {
+ die "line: {$line} code: {$code}" unless $code =~ /^[0-9A-F]+$/;
+ # single code point
+ $start = hex($code);
+ $end = hex($code);
+ }
+
+ if ($property eq "Alphabetic") {
+ push @alphabetic, {start => $start, end => $end};
+ }
+ elsif ($property eq "Lowercase") {
+ push @lowercase, {start => $start, end => $end};
+ }
+ elsif ($property eq "Uppercase") {
+ push @uppercase, {start => $start, end => $end};
+ }
+}
+
+my $num_category_ranges = scalar @category_ranges;
+my $num_alphabetic_ranges = scalar @alphabetic;
+my $num_lowercase_ranges = scalar @lowercase;
+my $num_uppercase_ranges = scalar @uppercase;
+my $num_white_space_ranges = scalar @white_space;
+my $num_hex_digit_ranges = scalar @hex_digits;
+my $num_join_control_ranges = scalar @join_control;
+
+# Start writing out the output file
open my $OT, '>', $output_table_file
or die "Could not open output file $output_table_file: $!\n";
-print $OT <<HEADER;
+print $OT <<"HEADER";
/*-------------------------------------------------------------------------
*
* unicode_category_table.h
@@ -188,11 +273,20 @@ typedef struct
uint8 category; /* General Category */
} pg_category_range;
-/* table of Unicode codepoint ranges and their categories */
-static const pg_category_range unicode_categories[$num_ranges] =
+typedef struct
{
+ uint32 first; /* Unicode codepoint */
+ uint32 last; /* Unicode codepoint */
+} pg_unicode_range;
+
HEADER
+print $OT <<"CATEGORY_TABLE";
+/* table of Unicode codepoint ranges and their categories */
+static const pg_category_range unicode_categories[$num_category_ranges] =
+{
+CATEGORY_TABLE
+
my $firsttime = 1;
foreach my $range (@category_ranges) {
printf $OT ",\n" unless $firsttime;
@@ -202,4 +296,101 @@ foreach my $range (@category_ranges) {
die "category missing: $range->{category}" unless $category;
printf $OT "\t{0x%06x, 0x%06x, %s}", $range->{start}, $range->{end}, $category;
}
+
+print $OT "\n};\n\n";
+
+print $OT <<"ALPHABETIC_TABLE";
+/* table of Unicode codepoint ranges of Alphabetic characters */
+static const pg_unicode_range unicode_alphabetic[$num_alphabetic_ranges] =
+{
+ALPHABETIC_TABLE
+
+$firsttime = 1;
+foreach my $range (@alphabetic) {
+ printf $OT ",\n" unless $firsttime;
+ $firsttime = 0;
+
+ printf $OT "\t{0x%06x, 0x%06x}", $range->{start}, $range->{end};
+}
+
+print $OT "\n};\n\n";
+
+print $OT <<"LOWERCASE_TABLE";
+/* table of Unicode codepoint ranges of Lowercase characters */
+static const pg_unicode_range unicode_lowercase[$num_lowercase_ranges] =
+{
+LOWERCASE_TABLE
+
+$firsttime = 1;
+foreach my $range (@lowercase) {
+ printf $OT ",\n" unless $firsttime;
+ $firsttime = 0;
+
+ printf $OT "\t{0x%06x, 0x%06x}", $range->{start}, $range->{end};
+}
+
+print $OT "\n};\n\n";
+
+print $OT <<"UPPERCASE_TABLE";
+/* table of Unicode codepoint ranges of Uppercase characters */
+static const pg_unicode_range unicode_uppercase[$num_uppercase_ranges] =
+{
+UPPERCASE_TABLE
+
+$firsttime = 1;
+foreach my $range (@uppercase) {
+ printf $OT ",\n" unless $firsttime;
+ $firsttime = 0;
+
+ printf $OT "\t{0x%06x, 0x%06x}", $range->{start}, $range->{end};
+}
+
+print $OT "\n};\n\n";
+
+print $OT <<"WHITE_SPACE_TABLE";
+/* table of Unicode codepoint ranges of White_Space characters */
+static const pg_unicode_range unicode_white_space[$num_white_space_ranges] =
+{
+WHITE_SPACE_TABLE
+
+$firsttime = 1;
+foreach my $range (@white_space) {
+ printf $OT ",\n" unless $firsttime;
+ $firsttime = 0;
+
+ printf $OT "\t{0x%06x, 0x%06x}", $range->{start}, $range->{end};
+}
+
+print $OT "\n};\n\n";
+
+print $OT <<"HEX_DIGITS_TABLE";
+/* table of Unicode codepoint ranges of Hex_Digit characters */
+static const pg_unicode_range unicode_hex_digit[$num_hex_digit_ranges] =
+{
+HEX_DIGITS_TABLE
+
+$firsttime = 1;
+foreach my $range (@hex_digits) {
+ printf $OT ",\n" unless $firsttime;
+ $firsttime = 0;
+
+ printf $OT "\t{0x%06x, 0x%06x}", $range->{start}, $range->{end};
+}
+
+print $OT "\n};\n\n";
+
+print $OT <<"JOIN_CONTROL_TABLE";
+/* table of Unicode codepoint ranges of Join_Control characters */
+static const pg_unicode_range unicode_join_control[$num_join_control_ranges] =
+{
+JOIN_CONTROL_TABLE
+
+$firsttime = 1;
+foreach my $range (@join_control) {
+ printf $OT ",\n" unless $firsttime;
+ $firsttime = 0;
+
+ printf $OT "\t{0x%06x, 0x%06x}", $range->{start}, $range->{end};
+}
+
print $OT "\n};\n";
diff --git a/src/common/unicode/meson.build b/src/common/unicode/meson.build
index e8cfdc1df4..3526ddb846 100644
--- a/src/common/unicode/meson.build
+++ b/src/common/unicode/meson.build
@@ -11,7 +11,7 @@ endif
# These files are part of the Unicode Character Database. Download them on
# demand.
-foreach f : ['CompositionExclusions.txt', 'DerivedNormalizationProps.txt', 'EastAsianWidth.txt', 'NormalizationTest.txt', 'UnicodeData.txt']
+foreach f : ['CompositionExclusions.txt', 'DerivedCoreProperties.txt', 'DerivedNormalizationProps.txt', 'EastAsianWidth.txt', 'NormalizationTest.txt', 'PropList.txt', 'UnicodeData.txt']
url = unicode_baseurl.format(UNICODE_VERSION, f)
target = custom_target(f,
output: f,
@@ -26,7 +26,7 @@ update_unicode_targets = []
update_unicode_targets += \
custom_target('unicode_category_table.h',
- input: [unicode_data['UnicodeData.txt']],
+ input: [unicode_data['UnicodeData.txt'], unicode_data['DerivedCoreProperties.txt'], unicode_data['PropList.txt']],
output: ['unicode_category_table.h'],
command: [
perl, files('generate-unicode_category_table.pl'),
diff --git a/src/common/unicode_category.c b/src/common/unicode_category.c
index 189cd6eca3..cfa086c42b 100644
--- a/src/common/unicode_category.c
+++ b/src/common/unicode_category.c
@@ -1,6 +1,8 @@
/*-------------------------------------------------------------------------
* unicode_category.c
- * Determine general category of Unicode characters.
+ * Determine general category and character class of Unicode
+ * characters. Encoding must be UTF8, where we assume that the pg_wchar
+ * representation is a code point.
*
* Portions Copyright (c) 2017-2023, PostgreSQL Global Development Group
*
@@ -18,24 +20,78 @@
#include "common/unicode_category.h"
#include "common/unicode_category_table.h"
+/*
+ * We use a mask word for convenience when testing for multiple categories at
+ * once. The number of Unicode General Categories should never grow, so a
+ * 32-bit mask is fine.
+ */
+#define PG_U_CATEGORY_MASK(X) ((uint32)(1 << (X)))
+
+#define PG_U_LU_MASK PG_U_CATEGORY_MASK(PG_U_UPPERCASE_LETTER)
+#define PG_U_LL_MASK PG_U_CATEGORY_MASK(PG_U_LOWERCASE_LETTER)
+#define PG_U_LT_MASK PG_U_CATEGORY_MASK(PG_U_TITLECASE_LETTER)
+#define PG_U_LC_MASK (PG_U_LU_MASK|PG_U_LL_MASK|PG_U_LT_MASK)
+#define PG_U_LM_MASK PG_U_CATEGORY_MASK(PG_U_MODIFIER_LETTER)
+#define PG_U_LO_MASK PG_U_CATEGORY_MASK(PG_U_OTHER_LETTER)
+#define PG_U_L_MASK (PG_U_LU_MASK|PG_U_LL_MASK|PG_U_LT_MASK|PG_U_LM_MASK|\
+ PG_U_LO_MASK)
+#define PG_U_MN_MASK PG_U_CATEGORY_MASK(PG_U_NONSPACING_MARK)
+#define PG_U_ME_MASK PG_U_CATEGORY_MASK(PG_U_ENCLOSING_MARK)
+#define PG_U_MC_MASK PG_U_CATEGORY_MASK(PG_U_SPACING_MARK)
+#define PG_U_M_MASK (PG_U_MN_MASK|PG_U_MC_MASK|PG_U_ME_MASK)
+#define PG_U_ND_MASK PG_U_CATEGORY_MASK(PG_U_DECIMAL_NUMBER)
+#define PG_U_NL_MASK PG_U_CATEGORY_MASK(PG_U_LETTER_NUMBER)
+#define PG_U_NO_MASK PG_U_CATEGORY_MASK(PG_U_OTHER_NUMBER)
+#define PG_U_N_MASK (PG_U_ND_MASK|PG_U_NL_MASK|PG_U_NO_MASK)
+#define PG_U_PC_MASK PG_U_CATEGORY_MASK(PG_U_CONNECTOR_PUNCTUATION)
+#define PG_U_PD_MASK PG_U_CATEGORY_MASK(PG_U_DASH_PUNCTUATION)
+#define PG_U_PS_MASK PG_U_CATEGORY_MASK(PG_U_OPEN_PUNCTUATION)
+#define PG_U_PE_MASK PG_U_CATEGORY_MASK(PG_U_CLOSE_PUNCTUATION)
+#define PG_U_PI_MASK PG_U_CATEGORY_MASK(PG_U_INITIAL_PUNCTUATION)
+#define PG_U_PF_MASK PG_U_CATEGORY_MASK(PG_U_FINAL_PUNCTUATION)
+#define PG_U_PO_MASK PG_U_CATEGORY_MASK(PG_U_OTHER_PUNCTUATION)
+#define PG_U_P_MASK (PG_U_PC_MASK|PG_U_PD_MASK|PG_U_PS_MASK|PG_U_PE_MASK|\
+ PG_U_PI_MASK|PG_U_PF_MASK|PG_U_PO_MASK)
+#define PG_U_SM_MASK PG_U_CATEGORY_MASK(PG_U_MATH_SYMBOL)
+#define PG_U_SC_MASK PG_U_CATEGORY_MASK(PG_U_CURRENCY_SYMBOL)
+#define PG_U_SK_MASK PG_U_CATEGORY_MASK(PG_U_MODIFIER_SYMBOL)
+#define PG_U_SO_MASK PG_U_CATEGORY_MASK(PG_U_OTHER_SYMBOL)
+#define PG_U_S_MASK (PG_U_SM_MASK|PG_U_SC_MASK|PG_U_SK_MASK|PG_U_SO_MASK)
+#define PG_U_ZS_MASK PG_U_CATEGORY_MASK(PG_U_SPACE_SEPARATOR)
+#define PG_U_ZL_MASK PG_U_CATEGORY_MASK(PG_U_LINE_SEPARATOR)
+#define PG_U_ZP_MASK PG_U_CATEGORY_MASK(PG_U_PARAGRAPH_SEPARATOR)
+#define PG_U_Z_MASK (PG_U_ZS_MASK|PG_U_ZL_MASK|PG_U_ZP_MASK)
+#define PG_U_CC_MASK PG_U_CATEGORY_MASK(PG_U_CONTROL)
+#define PG_U_CF_MASK PG_U_CATEGORY_MASK(PG_U_FORMAT)
+#define PG_U_CS_MASK PG_U_CATEGORY_MASK(PG_U_SURROGATE)
+#define PG_U_CO_MASK PG_U_CATEGORY_MASK(PG_U_PRIVATE_USE)
+#define PG_U_CN_MASK PG_U_CATEGORY_MASK(PG_U_UNASSIGNED)
+#define PG_U_C_MASK (PG_U_CC_MASK|PG_U_CF_MASK|PG_U_CS_MASK|PG_U_CO_MASK|\
+ PG_U_CN_MASK)
+
+#define PG_U_CHARACTER_TAB 0x09
+
+static bool range_search(const pg_unicode_range * tbl, Size size,
+ pg_wchar code);
+
/*
* Unicode general category for the given codepoint.
*/
pg_unicode_category
-unicode_category(pg_wchar ucs)
+unicode_category(pg_wchar code)
{
int min = 0;
int mid;
int max = lengthof(unicode_categories) - 1;
- Assert(ucs <= 0x10ffff);
+ Assert(code <= 0x10ffff);
while (max >= min)
{
mid = (min + max) / 2;
- if (ucs > unicode_categories[mid].last)
+ if (code > unicode_categories[mid].last)
min = mid + 1;
- else if (ucs < unicode_categories[mid].first)
+ else if (code < unicode_categories[mid].first)
max = mid - 1;
else
return unicode_categories[mid].category;
@@ -44,6 +100,171 @@ unicode_category(pg_wchar ucs)
return PG_U_UNASSIGNED;
}
+bool
+pg_u_prop_alphabetic(pg_wchar code)
+{
+ return range_search(unicode_alphabetic, lengthof(unicode_alphabetic),
+ code);
+}
+
+bool
+pg_u_prop_lowercase(pg_wchar code)
+{
+ return range_search(unicode_lowercase, lengthof(unicode_lowercase), code);
+}
+
+bool
+pg_u_prop_uppercase(pg_wchar code)
+{
+ return range_search(unicode_uppercase, lengthof(unicode_uppercase), code);
+}
+
+bool
+pg_u_prop_white_space(pg_wchar code)
+{
+ return range_search(unicode_white_space, lengthof(unicode_white_space),
+ code);
+}
+
+bool
+pg_u_prop_hex_digit(pg_wchar code)
+{
+ return range_search(unicode_hex_digit, lengthof(unicode_hex_digit), code);
+}
+
+bool
+pg_u_prop_join_control(pg_wchar code)
+{
+ return range_search(unicode_join_control, lengthof(unicode_join_control),
+ code);
+}
+
+/*
+ * The following functions implement the regex character classification as
+ * described at: http://www.unicode.org/reports/tr18/#Compatibility_Properties
+ *
+ * If 'posix' is true, implements the "POSIX Compatible" variant, otherwise
+ * the "Standard" variant.
+ */
+
+bool
+pg_u_isdigit(pg_wchar code, bool posix)
+{
+ if (posix)
+ return ('0' <= code && code <= '9');
+ else
+ return unicode_category(code) == PG_U_DECIMAL_NUMBER;
+}
+
+bool
+pg_u_isalpha(pg_wchar code)
+{
+ return pg_u_prop_alphabetic(code);
+}
+
+bool
+pg_u_isalnum(pg_wchar code, bool posix)
+{
+ return pg_u_isalpha(code) || pg_u_isdigit(code, posix);
+}
+
+bool
+pg_u_isword(pg_wchar code)
+{
+ uint32 category_mask = PG_U_CATEGORY_MASK(unicode_category(code));
+
+ return
+ category_mask & (PG_U_M_MASK|PG_U_ND_MASK|PG_U_PC_MASK) ||
+ pg_u_isalpha(code) ||
+ pg_u_prop_join_control(code);
+}
+
+bool
+pg_u_isupper(pg_wchar code)
+{
+ return pg_u_prop_uppercase(code);
+}
+
+bool
+pg_u_islower(pg_wchar code)
+{
+ return pg_u_prop_lowercase(code);
+}
+
+bool
+pg_u_isblank(pg_wchar code)
+{
+ return code == PG_U_CHARACTER_TAB ||
+ unicode_category(code) == PG_U_SPACE_SEPARATOR;
+}
+
+bool
+pg_u_iscntrl(pg_wchar code)
+{
+ return unicode_category(code) == PG_U_CONTROL;
+}
+
+bool
+pg_u_isgraph(pg_wchar code)
+{
+ uint32 category_mask = PG_U_CATEGORY_MASK(unicode_category(code));
+
+ if (category_mask & (PG_U_CC_MASK|PG_U_CS_MASK|PG_U_CN_MASK) ||
+ pg_u_isspace(code))
+ return false;
+ return true;
+}
+
+bool
+pg_u_isprint(pg_wchar code)
+{
+ pg_unicode_category category = unicode_category(code);
+
+ if (category == PG_U_CONTROL)
+ return false;
+
+ return pg_u_isgraph(code) || pg_u_isblank(code);
+}
+
+bool
+pg_u_ispunct(pg_wchar code, bool posix)
+{
+ uint32 category_mask;
+
+ if (posix)
+ {
+ if (pg_u_isalpha(code))
+ return false;
+
+ category_mask = PG_U_CATEGORY_MASK(unicode_category(code));
+ return category_mask & (PG_U_P_MASK|PG_U_S_MASK);
+ }
+ else
+ {
+ category_mask = PG_U_CATEGORY_MASK(unicode_category(code));
+
+ return category_mask & PG_U_P_MASK;
+ }
+}
+
+bool
+pg_u_isspace(pg_wchar code)
+{
+ return pg_u_prop_white_space(code);
+}
+
+bool
+pg_u_isxdigit(pg_wchar code, bool posix)
+{
+ if (posix)
+ return (('0' <= code && code <= '9') ||
+ ('A' <= code && code <= 'F') ||
+ ('a' <= code && code <= 'f'));
+ else
+ return unicode_category(code) == PG_U_DECIMAL_NUMBER ||
+ pg_u_prop_hex_digit(code);
+}
+
/*
* Description of Unicode general category.
*/
@@ -191,3 +412,30 @@ unicode_category_abbrev(pg_unicode_category category)
Assert(false);
return "??"; /* keep compiler quiet */
}
+
+/*
+ * Binary search to test if given codepoint exists in one of the ranges in the
+ * given table.
+ */
+static bool
+range_search(const pg_unicode_range * tbl, Size size, pg_wchar code)
+{
+ int min = 0;
+ int mid;
+ int max = size - 1;
+
+ Assert(code <= 0x10ffff);
+
+ while (max >= min)
+ {
+ mid = (min + max) / 2;
+ if (code > tbl[mid].last)
+ min = mid + 1;
+ else if (code < tbl[mid].first)
+ max = mid - 1;
+ else
+ return true;
+ }
+
+ return false;
+}
diff --git a/src/include/common/unicode_category.h b/src/include/common/unicode_category.h
index 81d38c7411..56fdc49802 100644
--- a/src/include/common/unicode_category.h
+++ b/src/include/common/unicode_category.h
@@ -62,7 +62,28 @@ typedef enum pg_unicode_category
} pg_unicode_category;
extern pg_unicode_category unicode_category(pg_wchar ucs);
-const char *unicode_category_string(pg_unicode_category category);
-const char *unicode_category_abbrev(pg_unicode_category category);
+extern const char *unicode_category_string(pg_unicode_category category);
+extern const char *unicode_category_abbrev(pg_unicode_category category);
+
+extern bool pg_u_prop_alphabetic(pg_wchar c);
+extern bool pg_u_prop_lowercase(pg_wchar c);
+extern bool pg_u_prop_uppercase(pg_wchar c);
+extern bool pg_u_prop_white_space(pg_wchar c);
+extern bool pg_u_prop_hex_digit(pg_wchar c);
+extern bool pg_u_prop_join_control(pg_wchar c);
+
+extern bool pg_u_isdigit(pg_wchar c, bool posix);
+extern bool pg_u_isalpha(pg_wchar c);
+extern bool pg_u_isalnum(pg_wchar c, bool posix);
+extern bool pg_u_isword(pg_wchar c);
+extern bool pg_u_isupper(pg_wchar c);
+extern bool pg_u_islower(pg_wchar c);
+extern bool pg_u_isblank(pg_wchar c);
+extern bool pg_u_iscntrl(pg_wchar c);
+extern bool pg_u_isgraph(pg_wchar c);
+extern bool pg_u_isprint(pg_wchar c);
+extern bool pg_u_ispunct(pg_wchar c, bool posix);
+extern bool pg_u_isspace(pg_wchar c);
+extern bool pg_u_isxdigit(pg_wchar c, bool posix);
#endif /* UNICODE_CATEGORY_H */
diff --git a/src/include/common/unicode_category_table.h b/src/include/common/unicode_category_table.h
index 14f1ea0677..86cdc9c0ed 100644
--- a/src/include/common/unicode_category_table.h
+++ b/src/include/common/unicode_category_table.h
@@ -25,6 +25,12 @@ typedef struct
uint8 category; /* General Category */
} pg_category_range;
+typedef struct
+{
+ uint32 first; /* Unicode codepoint */
+ uint32 last; /* Unicode codepoint */
+} pg_unicode_range;
+
/* table of Unicode codepoint ranges and their categories */
static const pg_category_range unicode_categories[3302] =
{
@@ -3331,3 +3337,2529 @@ static const pg_category_range unicode_categories[3302] =
{0x0f0000, 0x0ffffd, PG_U_PRIVATE_USE},
{0x100000, 0x10fffd, PG_U_PRIVATE_USE}
};
+
+/* table of Unicode codepoint ranges of Alphabetic characters */
+static const pg_unicode_range unicode_alphabetic[1141] =
+{
+ {0x000041, 0x00005a},
+ {0x000061, 0x00007a},
+ {0x0000aa, 0x0000aa},
+ {0x0000b5, 0x0000b5},
+ {0x0000ba, 0x0000ba},
+ {0x0000c0, 0x0000d6},
+ {0x0000d8, 0x0000f6},
+ {0x0000f8, 0x0001ba},
+ {0x0001bb, 0x0001bb},
+ {0x0001bc, 0x0001bf},
+ {0x0001c0, 0x0001c3},
+ {0x0001c4, 0x000293},
+ {0x000294, 0x000294},
+ {0x000295, 0x0002af},
+ {0x0002b0, 0x0002c1},
+ {0x0002c6, 0x0002d1},
+ {0x0002e0, 0x0002e4},
+ {0x0002ec, 0x0002ec},
+ {0x0002ee, 0x0002ee},
+ {0x000345, 0x000345},
+ {0x000370, 0x000373},
+ {0x000374, 0x000374},
+ {0x000376, 0x000377},
+ {0x00037a, 0x00037a},
+ {0x00037b, 0x00037d},
+ {0x00037f, 0x00037f},
+ {0x000386, 0x000386},
+ {0x000388, 0x00038a},
+ {0x00038c, 0x00038c},
+ {0x00038e, 0x0003a1},
+ {0x0003a3, 0x0003f5},
+ {0x0003f7, 0x000481},
+ {0x00048a, 0x00052f},
+ {0x000531, 0x000556},
+ {0x000559, 0x000559},
+ {0x000560, 0x000588},
+ {0x0005b0, 0x0005bd},
+ {0x0005bf, 0x0005bf},
+ {0x0005c1, 0x0005c2},
+ {0x0005c4, 0x0005c5},
+ {0x0005c7, 0x0005c7},
+ {0x0005d0, 0x0005ea},
+ {0x0005ef, 0x0005f2},
+ {0x000610, 0x00061a},
+ {0x000620, 0x00063f},
+ {0x000640, 0x000640},
+ {0x000641, 0x00064a},
+ {0x00064b, 0x000657},
+ {0x000659, 0x00065f},
+ {0x00066e, 0x00066f},
+ {0x000670, 0x000670},
+ {0x000671, 0x0006d3},
+ {0x0006d5, 0x0006d5},
+ {0x0006d6, 0x0006dc},
+ {0x0006e1, 0x0006e4},
+ {0x0006e5, 0x0006e6},
+ {0x0006e7, 0x0006e8},
+ {0x0006ed, 0x0006ed},
+ {0x0006ee, 0x0006ef},
+ {0x0006fa, 0x0006fc},
+ {0x0006ff, 0x0006ff},
+ {0x000710, 0x000710},
+ {0x000711, 0x000711},
+ {0x000712, 0x00072f},
+ {0x000730, 0x00073f},
+ {0x00074d, 0x0007a5},
+ {0x0007a6, 0x0007b0},
+ {0x0007b1, 0x0007b1},
+ {0x0007ca, 0x0007ea},
+ {0x0007f4, 0x0007f5},
+ {0x0007fa, 0x0007fa},
+ {0x000800, 0x000815},
+ {0x000816, 0x000817},
+ {0x00081a, 0x00081a},
+ {0x00081b, 0x000823},
+ {0x000824, 0x000824},
+ {0x000825, 0x000827},
+ {0x000828, 0x000828},
+ {0x000829, 0x00082c},
+ {0x000840, 0x000858},
+ {0x000860, 0x00086a},
+ {0x000870, 0x000887},
+ {0x000889, 0x00088e},
+ {0x0008a0, 0x0008c8},
+ {0x0008c9, 0x0008c9},
+ {0x0008d4, 0x0008df},
+ {0x0008e3, 0x0008e9},
+ {0x0008f0, 0x000902},
+ {0x000903, 0x000903},
+ {0x000904, 0x000939},
+ {0x00093a, 0x00093a},
+ {0x00093b, 0x00093b},
+ {0x00093d, 0x00093d},
+ {0x00093e, 0x000940},
+ {0x000941, 0x000948},
+ {0x000949, 0x00094c},
+ {0x00094e, 0x00094f},
+ {0x000950, 0x000950},
+ {0x000955, 0x000957},
+ {0x000958, 0x000961},
+ {0x000962, 0x000963},
+ {0x000971, 0x000971},
+ {0x000972, 0x000980},
+ {0x000981, 0x000981},
+ {0x000982, 0x000983},
+ {0x000985, 0x00098c},
+ {0x00098f, 0x000990},
+ {0x000993, 0x0009a8},
+ {0x0009aa, 0x0009b0},
+ {0x0009b2, 0x0009b2},
+ {0x0009b6, 0x0009b9},
+ {0x0009bd, 0x0009bd},
+ {0x0009be, 0x0009c0},
+ {0x0009c1, 0x0009c4},
+ {0x0009c7, 0x0009c8},
+ {0x0009cb, 0x0009cc},
+ {0x0009ce, 0x0009ce},
+ {0x0009d7, 0x0009d7},
+ {0x0009dc, 0x0009dd},
+ {0x0009df, 0x0009e1},
+ {0x0009e2, 0x0009e3},
+ {0x0009f0, 0x0009f1},
+ {0x0009fc, 0x0009fc},
+ {0x000a01, 0x000a02},
+ {0x000a03, 0x000a03},
+ {0x000a05, 0x000a0a},
+ {0x000a0f, 0x000a10},
+ {0x000a13, 0x000a28},
+ {0x000a2a, 0x000a30},
+ {0x000a32, 0x000a33},
+ {0x000a35, 0x000a36},
+ {0x000a38, 0x000a39},
+ {0x000a3e, 0x000a40},
+ {0x000a41, 0x000a42},
+ {0x000a47, 0x000a48},
+ {0x000a4b, 0x000a4c},
+ {0x000a51, 0x000a51},
+ {0x000a59, 0x000a5c},
+ {0x000a5e, 0x000a5e},
+ {0x000a70, 0x000a71},
+ {0x000a72, 0x000a74},
+ {0x000a75, 0x000a75},
+ {0x000a81, 0x000a82},
+ {0x000a83, 0x000a83},
+ {0x000a85, 0x000a8d},
+ {0x000a8f, 0x000a91},
+ {0x000a93, 0x000aa8},
+ {0x000aaa, 0x000ab0},
+ {0x000ab2, 0x000ab3},
+ {0x000ab5, 0x000ab9},
+ {0x000abd, 0x000abd},
+ {0x000abe, 0x000ac0},
+ {0x000ac1, 0x000ac5},
+ {0x000ac7, 0x000ac8},
+ {0x000ac9, 0x000ac9},
+ {0x000acb, 0x000acc},
+ {0x000ad0, 0x000ad0},
+ {0x000ae0, 0x000ae1},
+ {0x000ae2, 0x000ae3},
+ {0x000af9, 0x000af9},
+ {0x000afa, 0x000afc},
+ {0x000b01, 0x000b01},
+ {0x000b02, 0x000b03},
+ {0x000b05, 0x000b0c},
+ {0x000b0f, 0x000b10},
+ {0x000b13, 0x000b28},
+ {0x000b2a, 0x000b30},
+ {0x000b32, 0x000b33},
+ {0x000b35, 0x000b39},
+ {0x000b3d, 0x000b3d},
+ {0x000b3e, 0x000b3e},
+ {0x000b3f, 0x000b3f},
+ {0x000b40, 0x000b40},
+ {0x000b41, 0x000b44},
+ {0x000b47, 0x000b48},
+ {0x000b4b, 0x000b4c},
+ {0x000b56, 0x000b56},
+ {0x000b57, 0x000b57},
+ {0x000b5c, 0x000b5d},
+ {0x000b5f, 0x000b61},
+ {0x000b62, 0x000b63},
+ {0x000b71, 0x000b71},
+ {0x000b82, 0x000b82},
+ {0x000b83, 0x000b83},
+ {0x000b85, 0x000b8a},
+ {0x000b8e, 0x000b90},
+ {0x000b92, 0x000b95},
+ {0x000b99, 0x000b9a},
+ {0x000b9c, 0x000b9c},
+ {0x000b9e, 0x000b9f},
+ {0x000ba3, 0x000ba4},
+ {0x000ba8, 0x000baa},
+ {0x000bae, 0x000bb9},
+ {0x000bbe, 0x000bbf},
+ {0x000bc0, 0x000bc0},
+ {0x000bc1, 0x000bc2},
+ {0x000bc6, 0x000bc8},
+ {0x000bca, 0x000bcc},
+ {0x000bd0, 0x000bd0},
+ {0x000bd7, 0x000bd7},
+ {0x000c00, 0x000c00},
+ {0x000c01, 0x000c03},
+ {0x000c04, 0x000c04},
+ {0x000c05, 0x000c0c},
+ {0x000c0e, 0x000c10},
+ {0x000c12, 0x000c28},
+ {0x000c2a, 0x000c39},
+ {0x000c3d, 0x000c3d},
+ {0x000c3e, 0x000c40},
+ {0x000c41, 0x000c44},
+ {0x000c46, 0x000c48},
+ {0x000c4a, 0x000c4c},
+ {0x000c55, 0x000c56},
+ {0x000c58, 0x000c5a},
+ {0x000c5d, 0x000c5d},
+ {0x000c60, 0x000c61},
+ {0x000c62, 0x000c63},
+ {0x000c80, 0x000c80},
+ {0x000c81, 0x000c81},
+ {0x000c82, 0x000c83},
+ {0x000c85, 0x000c8c},
+ {0x000c8e, 0x000c90},
+ {0x000c92, 0x000ca8},
+ {0x000caa, 0x000cb3},
+ {0x000cb5, 0x000cb9},
+ {0x000cbd, 0x000cbd},
+ {0x000cbe, 0x000cbe},
+ {0x000cbf, 0x000cbf},
+ {0x000cc0, 0x000cc4},
+ {0x000cc6, 0x000cc6},
+ {0x000cc7, 0x000cc8},
+ {0x000cca, 0x000ccb},
+ {0x000ccc, 0x000ccc},
+ {0x000cd5, 0x000cd6},
+ {0x000cdd, 0x000cde},
+ {0x000ce0, 0x000ce1},
+ {0x000ce2, 0x000ce3},
+ {0x000cf1, 0x000cf2},
+ {0x000cf3, 0x000cf3},
+ {0x000d00, 0x000d01},
+ {0x000d02, 0x000d03},
+ {0x000d04, 0x000d0c},
+ {0x000d0e, 0x000d10},
+ {0x000d12, 0x000d3a},
+ {0x000d3d, 0x000d3d},
+ {0x000d3e, 0x000d40},
+ {0x000d41, 0x000d44},
+ {0x000d46, 0x000d48},
+ {0x000d4a, 0x000d4c},
+ {0x000d4e, 0x000d4e},
+ {0x000d54, 0x000d56},
+ {0x000d57, 0x000d57},
+ {0x000d5f, 0x000d61},
+ {0x000d62, 0x000d63},
+ {0x000d7a, 0x000d7f},
+ {0x000d81, 0x000d81},
+ {0x000d82, 0x000d83},
+ {0x000d85, 0x000d96},
+ {0x000d9a, 0x000db1},
+ {0x000db3, 0x000dbb},
+ {0x000dbd, 0x000dbd},
+ {0x000dc0, 0x000dc6},
+ {0x000dcf, 0x000dd1},
+ {0x000dd2, 0x000dd4},
+ {0x000dd6, 0x000dd6},
+ {0x000dd8, 0x000ddf},
+ {0x000df2, 0x000df3},
+ {0x000e01, 0x000e30},
+ {0x000e31, 0x000e31},
+ {0x000e32, 0x000e33},
+ {0x000e34, 0x000e3a},
+ {0x000e40, 0x000e45},
+ {0x000e46, 0x000e46},
+ {0x000e4d, 0x000e4d},
+ {0x000e81, 0x000e82},
+ {0x000e84, 0x000e84},
+ {0x000e86, 0x000e8a},
+ {0x000e8c, 0x000ea3},
+ {0x000ea5, 0x000ea5},
+ {0x000ea7, 0x000eb0},
+ {0x000eb1, 0x000eb1},
+ {0x000eb2, 0x000eb3},
+ {0x000eb4, 0x000eb9},
+ {0x000ebb, 0x000ebc},
+ {0x000ebd, 0x000ebd},
+ {0x000ec0, 0x000ec4},
+ {0x000ec6, 0x000ec6},
+ {0x000ecd, 0x000ecd},
+ {0x000edc, 0x000edf},
+ {0x000f00, 0x000f00},
+ {0x000f40, 0x000f47},
+ {0x000f49, 0x000f6c},
+ {0x000f71, 0x000f7e},
+ {0x000f7f, 0x000f7f},
+ {0x000f80, 0x000f83},
+ {0x000f88, 0x000f8c},
+ {0x000f8d, 0x000f97},
+ {0x000f99, 0x000fbc},
+ {0x001000, 0x00102a},
+ {0x00102b, 0x00102c},
+ {0x00102d, 0x001030},
+ {0x001031, 0x001031},
+ {0x001032, 0x001036},
+ {0x001038, 0x001038},
+ {0x00103b, 0x00103c},
+ {0x00103d, 0x00103e},
+ {0x00103f, 0x00103f},
+ {0x001050, 0x001055},
+ {0x001056, 0x001057},
+ {0x001058, 0x001059},
+ {0x00105a, 0x00105d},
+ {0x00105e, 0x001060},
+ {0x001061, 0x001061},
+ {0x001062, 0x001064},
+ {0x001065, 0x001066},
+ {0x001067, 0x00106d},
+ {0x00106e, 0x001070},
+ {0x001071, 0x001074},
+ {0x001075, 0x001081},
+ {0x001082, 0x001082},
+ {0x001083, 0x001084},
+ {0x001085, 0x001086},
+ {0x001087, 0x00108c},
+ {0x00108d, 0x00108d},
+ {0x00108e, 0x00108e},
+ {0x00108f, 0x00108f},
+ {0x00109a, 0x00109c},
+ {0x00109d, 0x00109d},
+ {0x0010a0, 0x0010c5},
+ {0x0010c7, 0x0010c7},
+ {0x0010cd, 0x0010cd},
+ {0x0010d0, 0x0010fa},
+ {0x0010fc, 0x0010fc},
+ {0x0010fd, 0x0010ff},
+ {0x001100, 0x001248},
+ {0x00124a, 0x00124d},
+ {0x001250, 0x001256},
+ {0x001258, 0x001258},
+ {0x00125a, 0x00125d},
+ {0x001260, 0x001288},
+ {0x00128a, 0x00128d},
+ {0x001290, 0x0012b0},
+ {0x0012b2, 0x0012b5},
+ {0x0012b8, 0x0012be},
+ {0x0012c0, 0x0012c0},
+ {0x0012c2, 0x0012c5},
+ {0x0012c8, 0x0012d6},
+ {0x0012d8, 0x001310},
+ {0x001312, 0x001315},
+ {0x001318, 0x00135a},
+ {0x001380, 0x00138f},
+ {0x0013a0, 0x0013f5},
+ {0x0013f8, 0x0013fd},
+ {0x001401, 0x00166c},
+ {0x00166f, 0x00167f},
+ {0x001681, 0x00169a},
+ {0x0016a0, 0x0016ea},
+ {0x0016ee, 0x0016f0},
+ {0x0016f1, 0x0016f8},
+ {0x001700, 0x001711},
+ {0x001712, 0x001713},
+ {0x00171f, 0x001731},
+ {0x001732, 0x001733},
+ {0x001740, 0x001751},
+ {0x001752, 0x001753},
+ {0x001760, 0x00176c},
+ {0x00176e, 0x001770},
+ {0x001772, 0x001773},
+ {0x001780, 0x0017b3},
+ {0x0017b6, 0x0017b6},
+ {0x0017b7, 0x0017bd},
+ {0x0017be, 0x0017c5},
+ {0x0017c6, 0x0017c6},
+ {0x0017c7, 0x0017c8},
+ {0x0017d7, 0x0017d7},
+ {0x0017dc, 0x0017dc},
+ {0x001820, 0x001842},
+ {0x001843, 0x001843},
+ {0x001844, 0x001878},
+ {0x001880, 0x001884},
+ {0x001885, 0x001886},
+ {0x001887, 0x0018a8},
+ {0x0018a9, 0x0018a9},
+ {0x0018aa, 0x0018aa},
+ {0x0018b0, 0x0018f5},
+ {0x001900, 0x00191e},
+ {0x001920, 0x001922},
+ {0x001923, 0x001926},
+ {0x001927, 0x001928},
+ {0x001929, 0x00192b},
+ {0x001930, 0x001931},
+ {0x001932, 0x001932},
+ {0x001933, 0x001938},
+ {0x001950, 0x00196d},
+ {0x001970, 0x001974},
+ {0x001980, 0x0019ab},
+ {0x0019b0, 0x0019c9},
+ {0x001a00, 0x001a16},
+ {0x001a17, 0x001a18},
+ {0x001a19, 0x001a1a},
+ {0x001a1b, 0x001a1b},
+ {0x001a20, 0x001a54},
+ {0x001a55, 0x001a55},
+ {0x001a56, 0x001a56},
+ {0x001a57, 0x001a57},
+ {0x001a58, 0x001a5e},
+ {0x001a61, 0x001a61},
+ {0x001a62, 0x001a62},
+ {0x001a63, 0x001a64},
+ {0x001a65, 0x001a6c},
+ {0x001a6d, 0x001a72},
+ {0x001a73, 0x001a74},
+ {0x001aa7, 0x001aa7},
+ {0x001abf, 0x001ac0},
+ {0x001acc, 0x001ace},
+ {0x001b00, 0x001b03},
+ {0x001b04, 0x001b04},
+ {0x001b05, 0x001b33},
+ {0x001b35, 0x001b35},
+ {0x001b36, 0x001b3a},
+ {0x001b3b, 0x001b3b},
+ {0x001b3c, 0x001b3c},
+ {0x001b3d, 0x001b41},
+ {0x001b42, 0x001b42},
+ {0x001b43, 0x001b43},
+ {0x001b45, 0x001b4c},
+ {0x001b80, 0x001b81},
+ {0x001b82, 0x001b82},
+ {0x001b83, 0x001ba0},
+ {0x001ba1, 0x001ba1},
+ {0x001ba2, 0x001ba5},
+ {0x001ba6, 0x001ba7},
+ {0x001ba8, 0x001ba9},
+ {0x001bac, 0x001bad},
+ {0x001bae, 0x001baf},
+ {0x001bba, 0x001be5},
+ {0x001be7, 0x001be7},
+ {0x001be8, 0x001be9},
+ {0x001bea, 0x001bec},
+ {0x001bed, 0x001bed},
+ {0x001bee, 0x001bee},
+ {0x001bef, 0x001bf1},
+ {0x001c00, 0x001c23},
+ {0x001c24, 0x001c2b},
+ {0x001c2c, 0x001c33},
+ {0x001c34, 0x001c35},
+ {0x001c36, 0x001c36},
+ {0x001c4d, 0x001c4f},
+ {0x001c5a, 0x001c77},
+ {0x001c78, 0x001c7d},
+ {0x001c80, 0x001c88},
+ {0x001c90, 0x001cba},
+ {0x001cbd, 0x001cbf},
+ {0x001ce9, 0x001cec},
+ {0x001cee, 0x001cf3},
+ {0x001cf5, 0x001cf6},
+ {0x001cfa, 0x001cfa},
+ {0x001d00, 0x001d2b},
+ {0x001d2c, 0x001d6a},
+ {0x001d6b, 0x001d77},
+ {0x001d78, 0x001d78},
+ {0x001d79, 0x001d9a},
+ {0x001d9b, 0x001dbf},
+ {0x001de7, 0x001df4},
+ {0x001e00, 0x001f15},
+ {0x001f18, 0x001f1d},
+ {0x001f20, 0x001f45},
+ {0x001f48, 0x001f4d},
+ {0x001f50, 0x001f57},
+ {0x001f59, 0x001f59},
+ {0x001f5b, 0x001f5b},
+ {0x001f5d, 0x001f5d},
+ {0x001f5f, 0x001f7d},
+ {0x001f80, 0x001fb4},
+ {0x001fb6, 0x001fbc},
+ {0x001fbe, 0x001fbe},
+ {0x001fc2, 0x001fc4},
+ {0x001fc6, 0x001fcc},
+ {0x001fd0, 0x001fd3},
+ {0x001fd6, 0x001fdb},
+ {0x001fe0, 0x001fec},
+ {0x001ff2, 0x001ff4},
+ {0x001ff6, 0x001ffc},
+ {0x002071, 0x002071},
+ {0x00207f, 0x00207f},
+ {0x002090, 0x00209c},
+ {0x002102, 0x002102},
+ {0x002107, 0x002107},
+ {0x00210a, 0x002113},
+ {0x002115, 0x002115},
+ {0x002119, 0x00211d},
+ {0x002124, 0x002124},
+ {0x002126, 0x002126},
+ {0x002128, 0x002128},
+ {0x00212a, 0x00212d},
+ {0x00212f, 0x002134},
+ {0x002135, 0x002138},
+ {0x002139, 0x002139},
+ {0x00213c, 0x00213f},
+ {0x002145, 0x002149},
+ {0x00214e, 0x00214e},
+ {0x002160, 0x002182},
+ {0x002183, 0x002184},
+ {0x002185, 0x002188},
+ {0x0024b6, 0x0024e9},
+ {0x002c00, 0x002c7b},
+ {0x002c7c, 0x002c7d},
+ {0x002c7e, 0x002ce4},
+ {0x002ceb, 0x002cee},
+ {0x002cf2, 0x002cf3},
+ {0x002d00, 0x002d25},
+ {0x002d27, 0x002d27},
+ {0x002d2d, 0x002d2d},
+ {0x002d30, 0x002d67},
+ {0x002d6f, 0x002d6f},
+ {0x002d80, 0x002d96},
+ {0x002da0, 0x002da6},
+ {0x002da8, 0x002dae},
+ {0x002db0, 0x002db6},
+ {0x002db8, 0x002dbe},
+ {0x002dc0, 0x002dc6},
+ {0x002dc8, 0x002dce},
+ {0x002dd0, 0x002dd6},
+ {0x002dd8, 0x002dde},
+ {0x002de0, 0x002dff},
+ {0x002e2f, 0x002e2f},
+ {0x003005, 0x003005},
+ {0x003006, 0x003006},
+ {0x003007, 0x003007},
+ {0x003021, 0x003029},
+ {0x003031, 0x003035},
+ {0x003038, 0x00303a},
+ {0x00303b, 0x00303b},
+ {0x00303c, 0x00303c},
+ {0x003041, 0x003096},
+ {0x00309d, 0x00309e},
+ {0x00309f, 0x00309f},
+ {0x0030a1, 0x0030fa},
+ {0x0030fc, 0x0030fe},
+ {0x0030ff, 0x0030ff},
+ {0x003105, 0x00312f},
+ {0x003131, 0x00318e},
+ {0x0031a0, 0x0031bf},
+ {0x0031f0, 0x0031ff},
+ {0x003400, 0x004dbf},
+ {0x004e00, 0x00a014},
+ {0x00a015, 0x00a015},
+ {0x00a016, 0x00a48c},
+ {0x00a4d0, 0x00a4f7},
+ {0x00a4f8, 0x00a4fd},
+ {0x00a500, 0x00a60b},
+ {0x00a60c, 0x00a60c},
+ {0x00a610, 0x00a61f},
+ {0x00a62a, 0x00a62b},
+ {0x00a640, 0x00a66d},
+ {0x00a66e, 0x00a66e},
+ {0x00a674, 0x00a67b},
+ {0x00a67f, 0x00a67f},
+ {0x00a680, 0x00a69b},
+ {0x00a69c, 0x00a69d},
+ {0x00a69e, 0x00a69f},
+ {0x00a6a0, 0x00a6e5},
+ {0x00a6e6, 0x00a6ef},
+ {0x00a717, 0x00a71f},
+ {0x00a722, 0x00a76f},
+ {0x00a770, 0x00a770},
+ {0x00a771, 0x00a787},
+ {0x00a788, 0x00a788},
+ {0x00a78b, 0x00a78e},
+ {0x00a78f, 0x00a78f},
+ {0x00a790, 0x00a7ca},
+ {0x00a7d0, 0x00a7d1},
+ {0x00a7d3, 0x00a7d3},
+ {0x00a7d5, 0x00a7d9},
+ {0x00a7f2, 0x00a7f4},
+ {0x00a7f5, 0x00a7f6},
+ {0x00a7f7, 0x00a7f7},
+ {0x00a7f8, 0x00a7f9},
+ {0x00a7fa, 0x00a7fa},
+ {0x00a7fb, 0x00a801},
+ {0x00a802, 0x00a802},
+ {0x00a803, 0x00a805},
+ {0x00a807, 0x00a80a},
+ {0x00a80b, 0x00a80b},
+ {0x00a80c, 0x00a822},
+ {0x00a823, 0x00a824},
+ {0x00a825, 0x00a826},
+ {0x00a827, 0x00a827},
+ {0x00a840, 0x00a873},
+ {0x00a880, 0x00a881},
+ {0x00a882, 0x00a8b3},
+ {0x00a8b4, 0x00a8c3},
+ {0x00a8c5, 0x00a8c5},
+ {0x00a8f2, 0x00a8f7},
+ {0x00a8fb, 0x00a8fb},
+ {0x00a8fd, 0x00a8fe},
+ {0x00a8ff, 0x00a8ff},
+ {0x00a90a, 0x00a925},
+ {0x00a926, 0x00a92a},
+ {0x00a930, 0x00a946},
+ {0x00a947, 0x00a951},
+ {0x00a952, 0x00a952},
+ {0x00a960, 0x00a97c},
+ {0x00a980, 0x00a982},
+ {0x00a983, 0x00a983},
+ {0x00a984, 0x00a9b2},
+ {0x00a9b4, 0x00a9b5},
+ {0x00a9b6, 0x00a9b9},
+ {0x00a9ba, 0x00a9bb},
+ {0x00a9bc, 0x00a9bd},
+ {0x00a9be, 0x00a9bf},
+ {0x00a9cf, 0x00a9cf},
+ {0x00a9e0, 0x00a9e4},
+ {0x00a9e5, 0x00a9e5},
+ {0x00a9e6, 0x00a9e6},
+ {0x00a9e7, 0x00a9ef},
+ {0x00a9fa, 0x00a9fe},
+ {0x00aa00, 0x00aa28},
+ {0x00aa29, 0x00aa2e},
+ {0x00aa2f, 0x00aa30},
+ {0x00aa31, 0x00aa32},
+ {0x00aa33, 0x00aa34},
+ {0x00aa35, 0x00aa36},
+ {0x00aa40, 0x00aa42},
+ {0x00aa43, 0x00aa43},
+ {0x00aa44, 0x00aa4b},
+ {0x00aa4c, 0x00aa4c},
+ {0x00aa4d, 0x00aa4d},
+ {0x00aa60, 0x00aa6f},
+ {0x00aa70, 0x00aa70},
+ {0x00aa71, 0x00aa76},
+ {0x00aa7a, 0x00aa7a},
+ {0x00aa7b, 0x00aa7b},
+ {0x00aa7c, 0x00aa7c},
+ {0x00aa7d, 0x00aa7d},
+ {0x00aa7e, 0x00aaaf},
+ {0x00aab0, 0x00aab0},
+ {0x00aab1, 0x00aab1},
+ {0x00aab2, 0x00aab4},
+ {0x00aab5, 0x00aab6},
+ {0x00aab7, 0x00aab8},
+ {0x00aab9, 0x00aabd},
+ {0x00aabe, 0x00aabe},
+ {0x00aac0, 0x00aac0},
+ {0x00aac2, 0x00aac2},
+ {0x00aadb, 0x00aadc},
+ {0x00aadd, 0x00aadd},
+ {0x00aae0, 0x00aaea},
+ {0x00aaeb, 0x00aaeb},
+ {0x00aaec, 0x00aaed},
+ {0x00aaee, 0x00aaef},
+ {0x00aaf2, 0x00aaf2},
+ {0x00aaf3, 0x00aaf4},
+ {0x00aaf5, 0x00aaf5},
+ {0x00ab01, 0x00ab06},
+ {0x00ab09, 0x00ab0e},
+ {0x00ab11, 0x00ab16},
+ {0x00ab20, 0x00ab26},
+ {0x00ab28, 0x00ab2e},
+ {0x00ab30, 0x00ab5a},
+ {0x00ab5c, 0x00ab5f},
+ {0x00ab60, 0x00ab68},
+ {0x00ab69, 0x00ab69},
+ {0x00ab70, 0x00abbf},
+ {0x00abc0, 0x00abe2},
+ {0x00abe3, 0x00abe4},
+ {0x00abe5, 0x00abe5},
+ {0x00abe6, 0x00abe7},
+ {0x00abe8, 0x00abe8},
+ {0x00abe9, 0x00abea},
+ {0x00ac00, 0x00d7a3},
+ {0x00d7b0, 0x00d7c6},
+ {0x00d7cb, 0x00d7fb},
+ {0x00f900, 0x00fa6d},
+ {0x00fa70, 0x00fad9},
+ {0x00fb00, 0x00fb06},
+ {0x00fb13, 0x00fb17},
+ {0x00fb1d, 0x00fb1d},
+ {0x00fb1e, 0x00fb1e},
+ {0x00fb1f, 0x00fb28},
+ {0x00fb2a, 0x00fb36},
+ {0x00fb38, 0x00fb3c},
+ {0x00fb3e, 0x00fb3e},
+ {0x00fb40, 0x00fb41},
+ {0x00fb43, 0x00fb44},
+ {0x00fb46, 0x00fbb1},
+ {0x00fbd3, 0x00fd3d},
+ {0x00fd50, 0x00fd8f},
+ {0x00fd92, 0x00fdc7},
+ {0x00fdf0, 0x00fdfb},
+ {0x00fe70, 0x00fe74},
+ {0x00fe76, 0x00fefc},
+ {0x00ff21, 0x00ff3a},
+ {0x00ff41, 0x00ff5a},
+ {0x00ff66, 0x00ff6f},
+ {0x00ff70, 0x00ff70},
+ {0x00ff71, 0x00ff9d},
+ {0x00ff9e, 0x00ff9f},
+ {0x00ffa0, 0x00ffbe},
+ {0x00ffc2, 0x00ffc7},
+ {0x00ffca, 0x00ffcf},
+ {0x00ffd2, 0x00ffd7},
+ {0x00ffda, 0x00ffdc},
+ {0x010000, 0x01000b},
+ {0x01000d, 0x010026},
+ {0x010028, 0x01003a},
+ {0x01003c, 0x01003d},
+ {0x01003f, 0x01004d},
+ {0x010050, 0x01005d},
+ {0x010080, 0x0100fa},
+ {0x010140, 0x010174},
+ {0x010280, 0x01029c},
+ {0x0102a0, 0x0102d0},
+ {0x010300, 0x01031f},
+ {0x01032d, 0x010340},
+ {0x010341, 0x010341},
+ {0x010342, 0x010349},
+ {0x01034a, 0x01034a},
+ {0x010350, 0x010375},
+ {0x010376, 0x01037a},
+ {0x010380, 0x01039d},
+ {0x0103a0, 0x0103c3},
+ {0x0103c8, 0x0103cf},
+ {0x0103d1, 0x0103d5},
+ {0x010400, 0x01044f},
+ {0x010450, 0x01049d},
+ {0x0104b0, 0x0104d3},
+ {0x0104d8, 0x0104fb},
+ {0x010500, 0x010527},
+ {0x010530, 0x010563},
+ {0x010570, 0x01057a},
+ {0x01057c, 0x01058a},
+ {0x01058c, 0x010592},
+ {0x010594, 0x010595},
+ {0x010597, 0x0105a1},
+ {0x0105a3, 0x0105b1},
+ {0x0105b3, 0x0105b9},
+ {0x0105bb, 0x0105bc},
+ {0x010600, 0x010736},
+ {0x010740, 0x010755},
+ {0x010760, 0x010767},
+ {0x010780, 0x010785},
+ {0x010787, 0x0107b0},
+ {0x0107b2, 0x0107ba},
+ {0x010800, 0x010805},
+ {0x010808, 0x010808},
+ {0x01080a, 0x010835},
+ {0x010837, 0x010838},
+ {0x01083c, 0x01083c},
+ {0x01083f, 0x010855},
+ {0x010860, 0x010876},
+ {0x010880, 0x01089e},
+ {0x0108e0, 0x0108f2},
+ {0x0108f4, 0x0108f5},
+ {0x010900, 0x010915},
+ {0x010920, 0x010939},
+ {0x010980, 0x0109b7},
+ {0x0109be, 0x0109bf},
+ {0x010a00, 0x010a00},
+ {0x010a01, 0x010a03},
+ {0x010a05, 0x010a06},
+ {0x010a0c, 0x010a0f},
+ {0x010a10, 0x010a13},
+ {0x010a15, 0x010a17},
+ {0x010a19, 0x010a35},
+ {0x010a60, 0x010a7c},
+ {0x010a80, 0x010a9c},
+ {0x010ac0, 0x010ac7},
+ {0x010ac9, 0x010ae4},
+ {0x010b00, 0x010b35},
+ {0x010b40, 0x010b55},
+ {0x010b60, 0x010b72},
+ {0x010b80, 0x010b91},
+ {0x010c00, 0x010c48},
+ {0x010c80, 0x010cb2},
+ {0x010cc0, 0x010cf2},
+ {0x010d00, 0x010d23},
+ {0x010d24, 0x010d27},
+ {0x010e80, 0x010ea9},
+ {0x010eab, 0x010eac},
+ {0x010eb0, 0x010eb1},
+ {0x010f00, 0x010f1c},
+ {0x010f27, 0x010f27},
+ {0x010f30, 0x010f45},
+ {0x010f70, 0x010f81},
+ {0x010fb0, 0x010fc4},
+ {0x010fe0, 0x010ff6},
+ {0x011000, 0x011000},
+ {0x011001, 0x011001},
+ {0x011002, 0x011002},
+ {0x011003, 0x011037},
+ {0x011038, 0x011045},
+ {0x011071, 0x011072},
+ {0x011073, 0x011074},
+ {0x011075, 0x011075},
+ {0x011080, 0x011081},
+ {0x011082, 0x011082},
+ {0x011083, 0x0110af},
+ {0x0110b0, 0x0110b2},
+ {0x0110b3, 0x0110b6},
+ {0x0110b7, 0x0110b8},
+ {0x0110c2, 0x0110c2},
+ {0x0110d0, 0x0110e8},
+ {0x011100, 0x011102},
+ {0x011103, 0x011126},
+ {0x011127, 0x01112b},
+ {0x01112c, 0x01112c},
+ {0x01112d, 0x011132},
+ {0x011144, 0x011144},
+ {0x011145, 0x011146},
+ {0x011147, 0x011147},
+ {0x011150, 0x011172},
+ {0x011176, 0x011176},
+ {0x011180, 0x011181},
+ {0x011182, 0x011182},
+ {0x011183, 0x0111b2},
+ {0x0111b3, 0x0111b5},
+ {0x0111b6, 0x0111be},
+ {0x0111bf, 0x0111bf},
+ {0x0111c1, 0x0111c4},
+ {0x0111ce, 0x0111ce},
+ {0x0111cf, 0x0111cf},
+ {0x0111da, 0x0111da},
+ {0x0111dc, 0x0111dc},
+ {0x011200, 0x011211},
+ {0x011213, 0x01122b},
+ {0x01122c, 0x01122e},
+ {0x01122f, 0x011231},
+ {0x011232, 0x011233},
+ {0x011234, 0x011234},
+ {0x011237, 0x011237},
+ {0x01123e, 0x01123e},
+ {0x01123f, 0x011240},
+ {0x011241, 0x011241},
+ {0x011280, 0x011286},
+ {0x011288, 0x011288},
+ {0x01128a, 0x01128d},
+ {0x01128f, 0x01129d},
+ {0x01129f, 0x0112a8},
+ {0x0112b0, 0x0112de},
+ {0x0112df, 0x0112df},
+ {0x0112e0, 0x0112e2},
+ {0x0112e3, 0x0112e8},
+ {0x011300, 0x011301},
+ {0x011302, 0x011303},
+ {0x011305, 0x01130c},
+ {0x01130f, 0x011310},
+ {0x011313, 0x011328},
+ {0x01132a, 0x011330},
+ {0x011332, 0x011333},
+ {0x011335, 0x011339},
+ {0x01133d, 0x01133d},
+ {0x01133e, 0x01133f},
+ {0x011340, 0x011340},
+ {0x011341, 0x011344},
+ {0x011347, 0x011348},
+ {0x01134b, 0x01134c},
+ {0x011350, 0x011350},
+ {0x011357, 0x011357},
+ {0x01135d, 0x011361},
+ {0x011362, 0x011363},
+ {0x011400, 0x011434},
+ {0x011435, 0x011437},
+ {0x011438, 0x01143f},
+ {0x011440, 0x011441},
+ {0x011443, 0x011444},
+ {0x011445, 0x011445},
+ {0x011447, 0x01144a},
+ {0x01145f, 0x011461},
+ {0x011480, 0x0114af},
+ {0x0114b0, 0x0114b2},
+ {0x0114b3, 0x0114b8},
+ {0x0114b9, 0x0114b9},
+ {0x0114ba, 0x0114ba},
+ {0x0114bb, 0x0114be},
+ {0x0114bf, 0x0114c0},
+ {0x0114c1, 0x0114c1},
+ {0x0114c4, 0x0114c5},
+ {0x0114c7, 0x0114c7},
+ {0x011580, 0x0115ae},
+ {0x0115af, 0x0115b1},
+ {0x0115b2, 0x0115b5},
+ {0x0115b8, 0x0115bb},
+ {0x0115bc, 0x0115bd},
+ {0x0115be, 0x0115be},
+ {0x0115d8, 0x0115db},
+ {0x0115dc, 0x0115dd},
+ {0x011600, 0x01162f},
+ {0x011630, 0x011632},
+ {0x011633, 0x01163a},
+ {0x01163b, 0x01163c},
+ {0x01163d, 0x01163d},
+ {0x01163e, 0x01163e},
+ {0x011640, 0x011640},
+ {0x011644, 0x011644},
+ {0x011680, 0x0116aa},
+ {0x0116ab, 0x0116ab},
+ {0x0116ac, 0x0116ac},
+ {0x0116ad, 0x0116ad},
+ {0x0116ae, 0x0116af},
+ {0x0116b0, 0x0116b5},
+ {0x0116b8, 0x0116b8},
+ {0x011700, 0x01171a},
+ {0x01171d, 0x01171f},
+ {0x011720, 0x011721},
+ {0x011722, 0x011725},
+ {0x011726, 0x011726},
+ {0x011727, 0x01172a},
+ {0x011740, 0x011746},
+ {0x011800, 0x01182b},
+ {0x01182c, 0x01182e},
+ {0x01182f, 0x011837},
+ {0x011838, 0x011838},
+ {0x0118a0, 0x0118df},
+ {0x0118ff, 0x011906},
+ {0x011909, 0x011909},
+ {0x01190c, 0x011913},
+ {0x011915, 0x011916},
+ {0x011918, 0x01192f},
+ {0x011930, 0x011935},
+ {0x011937, 0x011938},
+ {0x01193b, 0x01193c},
+ {0x01193f, 0x01193f},
+ {0x011940, 0x011940},
+ {0x011941, 0x011941},
+ {0x011942, 0x011942},
+ {0x0119a0, 0x0119a7},
+ {0x0119aa, 0x0119d0},
+ {0x0119d1, 0x0119d3},
+ {0x0119d4, 0x0119d7},
+ {0x0119da, 0x0119db},
+ {0x0119dc, 0x0119df},
+ {0x0119e1, 0x0119e1},
+ {0x0119e3, 0x0119e3},
+ {0x0119e4, 0x0119e4},
+ {0x011a00, 0x011a00},
+ {0x011a01, 0x011a0a},
+ {0x011a0b, 0x011a32},
+ {0x011a35, 0x011a38},
+ {0x011a39, 0x011a39},
+ {0x011a3a, 0x011a3a},
+ {0x011a3b, 0x011a3e},
+ {0x011a50, 0x011a50},
+ {0x011a51, 0x011a56},
+ {0x011a57, 0x011a58},
+ {0x011a59, 0x011a5b},
+ {0x011a5c, 0x011a89},
+ {0x011a8a, 0x011a96},
+ {0x011a97, 0x011a97},
+ {0x011a9d, 0x011a9d},
+ {0x011ab0, 0x011af8},
+ {0x011c00, 0x011c08},
+ {0x011c0a, 0x011c2e},
+ {0x011c2f, 0x011c2f},
+ {0x011c30, 0x011c36},
+ {0x011c38, 0x011c3d},
+ {0x011c3e, 0x011c3e},
+ {0x011c40, 0x011c40},
+ {0x011c72, 0x011c8f},
+ {0x011c92, 0x011ca7},
+ {0x011ca9, 0x011ca9},
+ {0x011caa, 0x011cb0},
+ {0x011cb1, 0x011cb1},
+ {0x011cb2, 0x011cb3},
+ {0x011cb4, 0x011cb4},
+ {0x011cb5, 0x011cb6},
+ {0x011d00, 0x011d06},
+ {0x011d08, 0x011d09},
+ {0x011d0b, 0x011d30},
+ {0x011d31, 0x011d36},
+ {0x011d3a, 0x011d3a},
+ {0x011d3c, 0x011d3d},
+ {0x011d3f, 0x011d41},
+ {0x011d43, 0x011d43},
+ {0x011d46, 0x011d46},
+ {0x011d47, 0x011d47},
+ {0x011d60, 0x011d65},
+ {0x011d67, 0x011d68},
+ {0x011d6a, 0x011d89},
+ {0x011d8a, 0x011d8e},
+ {0x011d90, 0x011d91},
+ {0x011d93, 0x011d94},
+ {0x011d95, 0x011d95},
+ {0x011d96, 0x011d96},
+ {0x011d98, 0x011d98},
+ {0x011ee0, 0x011ef2},
+ {0x011ef3, 0x011ef4},
+ {0x011ef5, 0x011ef6},
+ {0x011f00, 0x011f01},
+ {0x011f02, 0x011f02},
+ {0x011f03, 0x011f03},
+ {0x011f04, 0x011f10},
+ {0x011f12, 0x011f33},
+ {0x011f34, 0x011f35},
+ {0x011f36, 0x011f3a},
+ {0x011f3e, 0x011f3f},
+ {0x011f40, 0x011f40},
+ {0x011fb0, 0x011fb0},
+ {0x012000, 0x012399},
+ {0x012400, 0x01246e},
+ {0x012480, 0x012543},
+ {0x012f90, 0x012ff0},
+ {0x013000, 0x01342f},
+ {0x013441, 0x013446},
+ {0x014400, 0x014646},
+ {0x016800, 0x016a38},
+ {0x016a40, 0x016a5e},
+ {0x016a70, 0x016abe},
+ {0x016ad0, 0x016aed},
+ {0x016b00, 0x016b2f},
+ {0x016b40, 0x016b43},
+ {0x016b63, 0x016b77},
+ {0x016b7d, 0x016b8f},
+ {0x016e40, 0x016e7f},
+ {0x016f00, 0x016f4a},
+ {0x016f4f, 0x016f4f},
+ {0x016f50, 0x016f50},
+ {0x016f51, 0x016f87},
+ {0x016f8f, 0x016f92},
+ {0x016f93, 0x016f9f},
+ {0x016fe0, 0x016fe1},
+ {0x016fe3, 0x016fe3},
+ {0x016ff0, 0x016ff1},
+ {0x017000, 0x0187f7},
+ {0x018800, 0x018cd5},
+ {0x018d00, 0x018d08},
+ {0x01aff0, 0x01aff3},
+ {0x01aff5, 0x01affb},
+ {0x01affd, 0x01affe},
+ {0x01b000, 0x01b122},
+ {0x01b132, 0x01b132},
+ {0x01b150, 0x01b152},
+ {0x01b155, 0x01b155},
+ {0x01b164, 0x01b167},
+ {0x01b170, 0x01b2fb},
+ {0x01bc00, 0x01bc6a},
+ {0x01bc70, 0x01bc7c},
+ {0x01bc80, 0x01bc88},
+ {0x01bc90, 0x01bc99},
+ {0x01bc9e, 0x01bc9e},
+ {0x01d400, 0x01d454},
+ {0x01d456, 0x01d49c},
+ {0x01d49e, 0x01d49f},
+ {0x01d4a2, 0x01d4a2},
+ {0x01d4a5, 0x01d4a6},
+ {0x01d4a9, 0x01d4ac},
+ {0x01d4ae, 0x01d4b9},
+ {0x01d4bb, 0x01d4bb},
+ {0x01d4bd, 0x01d4c3},
+ {0x01d4c5, 0x01d505},
+ {0x01d507, 0x01d50a},
+ {0x01d50d, 0x01d514},
+ {0x01d516, 0x01d51c},
+ {0x01d51e, 0x01d539},
+ {0x01d53b, 0x01d53e},
+ {0x01d540, 0x01d544},
+ {0x01d546, 0x01d546},
+ {0x01d54a, 0x01d550},
+ {0x01d552, 0x01d6a5},
+ {0x01d6a8, 0x01d6c0},
+ {0x01d6c2, 0x01d6da},
+ {0x01d6dc, 0x01d6fa},
+ {0x01d6fc, 0x01d714},
+ {0x01d716, 0x01d734},
+ {0x01d736, 0x01d74e},
+ {0x01d750, 0x01d76e},
+ {0x01d770, 0x01d788},
+ {0x01d78a, 0x01d7a8},
+ {0x01d7aa, 0x01d7c2},
+ {0x01d7c4, 0x01d7cb},
+ {0x01df00, 0x01df09},
+ {0x01df0a, 0x01df0a},
+ {0x01df0b, 0x01df1e},
+ {0x01df25, 0x01df2a},
+ {0x01e000, 0x01e006},
+ {0x01e008, 0x01e018},
+ {0x01e01b, 0x01e021},
+ {0x01e023, 0x01e024},
+ {0x01e026, 0x01e02a},
+ {0x01e030, 0x01e06d},
+ {0x01e08f, 0x01e08f},
+ {0x01e100, 0x01e12c},
+ {0x01e137, 0x01e13d},
+ {0x01e14e, 0x01e14e},
+ {0x01e290, 0x01e2ad},
+ {0x01e2c0, 0x01e2eb},
+ {0x01e4d0, 0x01e4ea},
+ {0x01e4eb, 0x01e4eb},
+ {0x01e7e0, 0x01e7e6},
+ {0x01e7e8, 0x01e7eb},
+ {0x01e7ed, 0x01e7ee},
+ {0x01e7f0, 0x01e7fe},
+ {0x01e800, 0x01e8c4},
+ {0x01e900, 0x01e943},
+ {0x01e947, 0x01e947},
+ {0x01e94b, 0x01e94b},
+ {0x01ee00, 0x01ee03},
+ {0x01ee05, 0x01ee1f},
+ {0x01ee21, 0x01ee22},
+ {0x01ee24, 0x01ee24},
+ {0x01ee27, 0x01ee27},
+ {0x01ee29, 0x01ee32},
+ {0x01ee34, 0x01ee37},
+ {0x01ee39, 0x01ee39},
+ {0x01ee3b, 0x01ee3b},
+ {0x01ee42, 0x01ee42},
+ {0x01ee47, 0x01ee47},
+ {0x01ee49, 0x01ee49},
+ {0x01ee4b, 0x01ee4b},
+ {0x01ee4d, 0x01ee4f},
+ {0x01ee51, 0x01ee52},
+ {0x01ee54, 0x01ee54},
+ {0x01ee57, 0x01ee57},
+ {0x01ee59, 0x01ee59},
+ {0x01ee5b, 0x01ee5b},
+ {0x01ee5d, 0x01ee5d},
+ {0x01ee5f, 0x01ee5f},
+ {0x01ee61, 0x01ee62},
+ {0x01ee64, 0x01ee64},
+ {0x01ee67, 0x01ee6a},
+ {0x01ee6c, 0x01ee72},
+ {0x01ee74, 0x01ee77},
+ {0x01ee79, 0x01ee7c},
+ {0x01ee7e, 0x01ee7e},
+ {0x01ee80, 0x01ee89},
+ {0x01ee8b, 0x01ee9b},
+ {0x01eea1, 0x01eea3},
+ {0x01eea5, 0x01eea9},
+ {0x01eeab, 0x01eebb},
+ {0x01f130, 0x01f149},
+ {0x01f150, 0x01f169},
+ {0x01f170, 0x01f189},
+ {0x020000, 0x02a6df},
+ {0x02a700, 0x02b739},
+ {0x02b740, 0x02b81d},
+ {0x02b820, 0x02cea1},
+ {0x02ceb0, 0x02ebe0},
+ {0x02ebf0, 0x02ee5d},
+ {0x02f800, 0x02fa1d},
+ {0x030000, 0x03134a},
+ {0x031350, 0x0323af}
+};
+
+/* table of Unicode codepoint ranges of Lowercase characters */
+static const pg_unicode_range unicode_lowercase[686] =
+{
+ {0x000061, 0x00007a},
+ {0x0000aa, 0x0000aa},
+ {0x0000b5, 0x0000b5},
+ {0x0000ba, 0x0000ba},
+ {0x0000df, 0x0000f6},
+ {0x0000f8, 0x0000ff},
+ {0x000101, 0x000101},
+ {0x000103, 0x000103},
+ {0x000105, 0x000105},
+ {0x000107, 0x000107},
+ {0x000109, 0x000109},
+ {0x00010b, 0x00010b},
+ {0x00010d, 0x00010d},
+ {0x00010f, 0x00010f},
+ {0x000111, 0x000111},
+ {0x000113, 0x000113},
+ {0x000115, 0x000115},
+ {0x000117, 0x000117},
+ {0x000119, 0x000119},
+ {0x00011b, 0x00011b},
+ {0x00011d, 0x00011d},
+ {0x00011f, 0x00011f},
+ {0x000121, 0x000121},
+ {0x000123, 0x000123},
+ {0x000125, 0x000125},
+ {0x000127, 0x000127},
+ {0x000129, 0x000129},
+ {0x00012b, 0x00012b},
+ {0x00012d, 0x00012d},
+ {0x00012f, 0x00012f},
+ {0x000131, 0x000131},
+ {0x000133, 0x000133},
+ {0x000135, 0x000135},
+ {0x000137, 0x000138},
+ {0x00013a, 0x00013a},
+ {0x00013c, 0x00013c},
+ {0x00013e, 0x00013e},
+ {0x000140, 0x000140},
+ {0x000142, 0x000142},
+ {0x000144, 0x000144},
+ {0x000146, 0x000146},
+ {0x000148, 0x000149},
+ {0x00014b, 0x00014b},
+ {0x00014d, 0x00014d},
+ {0x00014f, 0x00014f},
+ {0x000151, 0x000151},
+ {0x000153, 0x000153},
+ {0x000155, 0x000155},
+ {0x000157, 0x000157},
+ {0x000159, 0x000159},
+ {0x00015b, 0x00015b},
+ {0x00015d, 0x00015d},
+ {0x00015f, 0x00015f},
+ {0x000161, 0x000161},
+ {0x000163, 0x000163},
+ {0x000165, 0x000165},
+ {0x000167, 0x000167},
+ {0x000169, 0x000169},
+ {0x00016b, 0x00016b},
+ {0x00016d, 0x00016d},
+ {0x00016f, 0x00016f},
+ {0x000171, 0x000171},
+ {0x000173, 0x000173},
+ {0x000175, 0x000175},
+ {0x000177, 0x000177},
+ {0x00017a, 0x00017a},
+ {0x00017c, 0x00017c},
+ {0x00017e, 0x000180},
+ {0x000183, 0x000183},
+ {0x000185, 0x000185},
+ {0x000188, 0x000188},
+ {0x00018c, 0x00018d},
+ {0x000192, 0x000192},
+ {0x000195, 0x000195},
+ {0x000199, 0x00019b},
+ {0x00019e, 0x00019e},
+ {0x0001a1, 0x0001a1},
+ {0x0001a3, 0x0001a3},
+ {0x0001a5, 0x0001a5},
+ {0x0001a8, 0x0001a8},
+ {0x0001aa, 0x0001ab},
+ {0x0001ad, 0x0001ad},
+ {0x0001b0, 0x0001b0},
+ {0x0001b4, 0x0001b4},
+ {0x0001b6, 0x0001b6},
+ {0x0001b9, 0x0001ba},
+ {0x0001bd, 0x0001bf},
+ {0x0001c6, 0x0001c6},
+ {0x0001c9, 0x0001c9},
+ {0x0001cc, 0x0001cc},
+ {0x0001ce, 0x0001ce},
+ {0x0001d0, 0x0001d0},
+ {0x0001d2, 0x0001d2},
+ {0x0001d4, 0x0001d4},
+ {0x0001d6, 0x0001d6},
+ {0x0001d8, 0x0001d8},
+ {0x0001da, 0x0001da},
+ {0x0001dc, 0x0001dd},
+ {0x0001df, 0x0001df},
+ {0x0001e1, 0x0001e1},
+ {0x0001e3, 0x0001e3},
+ {0x0001e5, 0x0001e5},
+ {0x0001e7, 0x0001e7},
+ {0x0001e9, 0x0001e9},
+ {0x0001eb, 0x0001eb},
+ {0x0001ed, 0x0001ed},
+ {0x0001ef, 0x0001f0},
+ {0x0001f3, 0x0001f3},
+ {0x0001f5, 0x0001f5},
+ {0x0001f9, 0x0001f9},
+ {0x0001fb, 0x0001fb},
+ {0x0001fd, 0x0001fd},
+ {0x0001ff, 0x0001ff},
+ {0x000201, 0x000201},
+ {0x000203, 0x000203},
+ {0x000205, 0x000205},
+ {0x000207, 0x000207},
+ {0x000209, 0x000209},
+ {0x00020b, 0x00020b},
+ {0x00020d, 0x00020d},
+ {0x00020f, 0x00020f},
+ {0x000211, 0x000211},
+ {0x000213, 0x000213},
+ {0x000215, 0x000215},
+ {0x000217, 0x000217},
+ {0x000219, 0x000219},
+ {0x00021b, 0x00021b},
+ {0x00021d, 0x00021d},
+ {0x00021f, 0x00021f},
+ {0x000221, 0x000221},
+ {0x000223, 0x000223},
+ {0x000225, 0x000225},
+ {0x000227, 0x000227},
+ {0x000229, 0x000229},
+ {0x00022b, 0x00022b},
+ {0x00022d, 0x00022d},
+ {0x00022f, 0x00022f},
+ {0x000231, 0x000231},
+ {0x000233, 0x000239},
+ {0x00023c, 0x00023c},
+ {0x00023f, 0x000240},
+ {0x000242, 0x000242},
+ {0x000247, 0x000247},
+ {0x000249, 0x000249},
+ {0x00024b, 0x00024b},
+ {0x00024d, 0x00024d},
+ {0x00024f, 0x000293},
+ {0x000295, 0x0002af},
+ {0x0002b0, 0x0002b8},
+ {0x0002c0, 0x0002c1},
+ {0x0002e0, 0x0002e4},
+ {0x000345, 0x000345},
+ {0x000371, 0x000371},
+ {0x000373, 0x000373},
+ {0x000377, 0x000377},
+ {0x00037a, 0x00037a},
+ {0x00037b, 0x00037d},
+ {0x000390, 0x000390},
+ {0x0003ac, 0x0003ce},
+ {0x0003d0, 0x0003d1},
+ {0x0003d5, 0x0003d7},
+ {0x0003d9, 0x0003d9},
+ {0x0003db, 0x0003db},
+ {0x0003dd, 0x0003dd},
+ {0x0003df, 0x0003df},
+ {0x0003e1, 0x0003e1},
+ {0x0003e3, 0x0003e3},
+ {0x0003e5, 0x0003e5},
+ {0x0003e7, 0x0003e7},
+ {0x0003e9, 0x0003e9},
+ {0x0003eb, 0x0003eb},
+ {0x0003ed, 0x0003ed},
+ {0x0003ef, 0x0003f3},
+ {0x0003f5, 0x0003f5},
+ {0x0003f8, 0x0003f8},
+ {0x0003fb, 0x0003fc},
+ {0x000430, 0x00045f},
+ {0x000461, 0x000461},
+ {0x000463, 0x000463},
+ {0x000465, 0x000465},
+ {0x000467, 0x000467},
+ {0x000469, 0x000469},
+ {0x00046b, 0x00046b},
+ {0x00046d, 0x00046d},
+ {0x00046f, 0x00046f},
+ {0x000471, 0x000471},
+ {0x000473, 0x000473},
+ {0x000475, 0x000475},
+ {0x000477, 0x000477},
+ {0x000479, 0x000479},
+ {0x00047b, 0x00047b},
+ {0x00047d, 0x00047d},
+ {0x00047f, 0x00047f},
+ {0x000481, 0x000481},
+ {0x00048b, 0x00048b},
+ {0x00048d, 0x00048d},
+ {0x00048f, 0x00048f},
+ {0x000491, 0x000491},
+ {0x000493, 0x000493},
+ {0x000495, 0x000495},
+ {0x000497, 0x000497},
+ {0x000499, 0x000499},
+ {0x00049b, 0x00049b},
+ {0x00049d, 0x00049d},
+ {0x00049f, 0x00049f},
+ {0x0004a1, 0x0004a1},
+ {0x0004a3, 0x0004a3},
+ {0x0004a5, 0x0004a5},
+ {0x0004a7, 0x0004a7},
+ {0x0004a9, 0x0004a9},
+ {0x0004ab, 0x0004ab},
+ {0x0004ad, 0x0004ad},
+ {0x0004af, 0x0004af},
+ {0x0004b1, 0x0004b1},
+ {0x0004b3, 0x0004b3},
+ {0x0004b5, 0x0004b5},
+ {0x0004b7, 0x0004b7},
+ {0x0004b9, 0x0004b9},
+ {0x0004bb, 0x0004bb},
+ {0x0004bd, 0x0004bd},
+ {0x0004bf, 0x0004bf},
+ {0x0004c2, 0x0004c2},
+ {0x0004c4, 0x0004c4},
+ {0x0004c6, 0x0004c6},
+ {0x0004c8, 0x0004c8},
+ {0x0004ca, 0x0004ca},
+ {0x0004cc, 0x0004cc},
+ {0x0004ce, 0x0004cf},
+ {0x0004d1, 0x0004d1},
+ {0x0004d3, 0x0004d3},
+ {0x0004d5, 0x0004d5},
+ {0x0004d7, 0x0004d7},
+ {0x0004d9, 0x0004d9},
+ {0x0004db, 0x0004db},
+ {0x0004dd, 0x0004dd},
+ {0x0004df, 0x0004df},
+ {0x0004e1, 0x0004e1},
+ {0x0004e3, 0x0004e3},
+ {0x0004e5, 0x0004e5},
+ {0x0004e7, 0x0004e7},
+ {0x0004e9, 0x0004e9},
+ {0x0004eb, 0x0004eb},
+ {0x0004ed, 0x0004ed},
+ {0x0004ef, 0x0004ef},
+ {0x0004f1, 0x0004f1},
+ {0x0004f3, 0x0004f3},
+ {0x0004f5, 0x0004f5},
+ {0x0004f7, 0x0004f7},
+ {0x0004f9, 0x0004f9},
+ {0x0004fb, 0x0004fb},
+ {0x0004fd, 0x0004fd},
+ {0x0004ff, 0x0004ff},
+ {0x000501, 0x000501},
+ {0x000503, 0x000503},
+ {0x000505, 0x000505},
+ {0x000507, 0x000507},
+ {0x000509, 0x000509},
+ {0x00050b, 0x00050b},
+ {0x00050d, 0x00050d},
+ {0x00050f, 0x00050f},
+ {0x000511, 0x000511},
+ {0x000513, 0x000513},
+ {0x000515, 0x000515},
+ {0x000517, 0x000517},
+ {0x000519, 0x000519},
+ {0x00051b, 0x00051b},
+ {0x00051d, 0x00051d},
+ {0x00051f, 0x00051f},
+ {0x000521, 0x000521},
+ {0x000523, 0x000523},
+ {0x000525, 0x000525},
+ {0x000527, 0x000527},
+ {0x000529, 0x000529},
+ {0x00052b, 0x00052b},
+ {0x00052d, 0x00052d},
+ {0x00052f, 0x00052f},
+ {0x000560, 0x000588},
+ {0x0010d0, 0x0010fa},
+ {0x0010fc, 0x0010fc},
+ {0x0010fd, 0x0010ff},
+ {0x0013f8, 0x0013fd},
+ {0x001c80, 0x001c88},
+ {0x001d00, 0x001d2b},
+ {0x001d2c, 0x001d6a},
+ {0x001d6b, 0x001d77},
+ {0x001d78, 0x001d78},
+ {0x001d79, 0x001d9a},
+ {0x001d9b, 0x001dbf},
+ {0x001e01, 0x001e01},
+ {0x001e03, 0x001e03},
+ {0x001e05, 0x001e05},
+ {0x001e07, 0x001e07},
+ {0x001e09, 0x001e09},
+ {0x001e0b, 0x001e0b},
+ {0x001e0d, 0x001e0d},
+ {0x001e0f, 0x001e0f},
+ {0x001e11, 0x001e11},
+ {0x001e13, 0x001e13},
+ {0x001e15, 0x001e15},
+ {0x001e17, 0x001e17},
+ {0x001e19, 0x001e19},
+ {0x001e1b, 0x001e1b},
+ {0x001e1d, 0x001e1d},
+ {0x001e1f, 0x001e1f},
+ {0x001e21, 0x001e21},
+ {0x001e23, 0x001e23},
+ {0x001e25, 0x001e25},
+ {0x001e27, 0x001e27},
+ {0x001e29, 0x001e29},
+ {0x001e2b, 0x001e2b},
+ {0x001e2d, 0x001e2d},
+ {0x001e2f, 0x001e2f},
+ {0x001e31, 0x001e31},
+ {0x001e33, 0x001e33},
+ {0x001e35, 0x001e35},
+ {0x001e37, 0x001e37},
+ {0x001e39, 0x001e39},
+ {0x001e3b, 0x001e3b},
+ {0x001e3d, 0x001e3d},
+ {0x001e3f, 0x001e3f},
+ {0x001e41, 0x001e41},
+ {0x001e43, 0x001e43},
+ {0x001e45, 0x001e45},
+ {0x001e47, 0x001e47},
+ {0x001e49, 0x001e49},
+ {0x001e4b, 0x001e4b},
+ {0x001e4d, 0x001e4d},
+ {0x001e4f, 0x001e4f},
+ {0x001e51, 0x001e51},
+ {0x001e53, 0x001e53},
+ {0x001e55, 0x001e55},
+ {0x001e57, 0x001e57},
+ {0x001e59, 0x001e59},
+ {0x001e5b, 0x001e5b},
+ {0x001e5d, 0x001e5d},
+ {0x001e5f, 0x001e5f},
+ {0x001e61, 0x001e61},
+ {0x001e63, 0x001e63},
+ {0x001e65, 0x001e65},
+ {0x001e67, 0x001e67},
+ {0x001e69, 0x001e69},
+ {0x001e6b, 0x001e6b},
+ {0x001e6d, 0x001e6d},
+ {0x001e6f, 0x001e6f},
+ {0x001e71, 0x001e71},
+ {0x001e73, 0x001e73},
+ {0x001e75, 0x001e75},
+ {0x001e77, 0x001e77},
+ {0x001e79, 0x001e79},
+ {0x001e7b, 0x001e7b},
+ {0x001e7d, 0x001e7d},
+ {0x001e7f, 0x001e7f},
+ {0x001e81, 0x001e81},
+ {0x001e83, 0x001e83},
+ {0x001e85, 0x001e85},
+ {0x001e87, 0x001e87},
+ {0x001e89, 0x001e89},
+ {0x001e8b, 0x001e8b},
+ {0x001e8d, 0x001e8d},
+ {0x001e8f, 0x001e8f},
+ {0x001e91, 0x001e91},
+ {0x001e93, 0x001e93},
+ {0x001e95, 0x001e9d},
+ {0x001e9f, 0x001e9f},
+ {0x001ea1, 0x001ea1},
+ {0x001ea3, 0x001ea3},
+ {0x001ea5, 0x001ea5},
+ {0x001ea7, 0x001ea7},
+ {0x001ea9, 0x001ea9},
+ {0x001eab, 0x001eab},
+ {0x001ead, 0x001ead},
+ {0x001eaf, 0x001eaf},
+ {0x001eb1, 0x001eb1},
+ {0x001eb3, 0x001eb3},
+ {0x001eb5, 0x001eb5},
+ {0x001eb7, 0x001eb7},
+ {0x001eb9, 0x001eb9},
+ {0x001ebb, 0x001ebb},
+ {0x001ebd, 0x001ebd},
+ {0x001ebf, 0x001ebf},
+ {0x001ec1, 0x001ec1},
+ {0x001ec3, 0x001ec3},
+ {0x001ec5, 0x001ec5},
+ {0x001ec7, 0x001ec7},
+ {0x001ec9, 0x001ec9},
+ {0x001ecb, 0x001ecb},
+ {0x001ecd, 0x001ecd},
+ {0x001ecf, 0x001ecf},
+ {0x001ed1, 0x001ed1},
+ {0x001ed3, 0x001ed3},
+ {0x001ed5, 0x001ed5},
+ {0x001ed7, 0x001ed7},
+ {0x001ed9, 0x001ed9},
+ {0x001edb, 0x001edb},
+ {0x001edd, 0x001edd},
+ {0x001edf, 0x001edf},
+ {0x001ee1, 0x001ee1},
+ {0x001ee3, 0x001ee3},
+ {0x001ee5, 0x001ee5},
+ {0x001ee7, 0x001ee7},
+ {0x001ee9, 0x001ee9},
+ {0x001eeb, 0x001eeb},
+ {0x001eed, 0x001eed},
+ {0x001eef, 0x001eef},
+ {0x001ef1, 0x001ef1},
+ {0x001ef3, 0x001ef3},
+ {0x001ef5, 0x001ef5},
+ {0x001ef7, 0x001ef7},
+ {0x001ef9, 0x001ef9},
+ {0x001efb, 0x001efb},
+ {0x001efd, 0x001efd},
+ {0x001eff, 0x001f07},
+ {0x001f10, 0x001f15},
+ {0x001f20, 0x001f27},
+ {0x001f30, 0x001f37},
+ {0x001f40, 0x001f45},
+ {0x001f50, 0x001f57},
+ {0x001f60, 0x001f67},
+ {0x001f70, 0x001f7d},
+ {0x001f80, 0x001f87},
+ {0x001f90, 0x001f97},
+ {0x001fa0, 0x001fa7},
+ {0x001fb0, 0x001fb4},
+ {0x001fb6, 0x001fb7},
+ {0x001fbe, 0x001fbe},
+ {0x001fc2, 0x001fc4},
+ {0x001fc6, 0x001fc7},
+ {0x001fd0, 0x001fd3},
+ {0x001fd6, 0x001fd7},
+ {0x001fe0, 0x001fe7},
+ {0x001ff2, 0x001ff4},
+ {0x001ff6, 0x001ff7},
+ {0x002071, 0x002071},
+ {0x00207f, 0x00207f},
+ {0x002090, 0x00209c},
+ {0x00210a, 0x00210a},
+ {0x00210e, 0x00210f},
+ {0x002113, 0x002113},
+ {0x00212f, 0x00212f},
+ {0x002134, 0x002134},
+ {0x002139, 0x002139},
+ {0x00213c, 0x00213d},
+ {0x002146, 0x002149},
+ {0x00214e, 0x00214e},
+ {0x002170, 0x00217f},
+ {0x002184, 0x002184},
+ {0x0024d0, 0x0024e9},
+ {0x002c30, 0x002c5f},
+ {0x002c61, 0x002c61},
+ {0x002c65, 0x002c66},
+ {0x002c68, 0x002c68},
+ {0x002c6a, 0x002c6a},
+ {0x002c6c, 0x002c6c},
+ {0x002c71, 0x002c71},
+ {0x002c73, 0x002c74},
+ {0x002c76, 0x002c7b},
+ {0x002c7c, 0x002c7d},
+ {0x002c81, 0x002c81},
+ {0x002c83, 0x002c83},
+ {0x002c85, 0x002c85},
+ {0x002c87, 0x002c87},
+ {0x002c89, 0x002c89},
+ {0x002c8b, 0x002c8b},
+ {0x002c8d, 0x002c8d},
+ {0x002c8f, 0x002c8f},
+ {0x002c91, 0x002c91},
+ {0x002c93, 0x002c93},
+ {0x002c95, 0x002c95},
+ {0x002c97, 0x002c97},
+ {0x002c99, 0x002c99},
+ {0x002c9b, 0x002c9b},
+ {0x002c9d, 0x002c9d},
+ {0x002c9f, 0x002c9f},
+ {0x002ca1, 0x002ca1},
+ {0x002ca3, 0x002ca3},
+ {0x002ca5, 0x002ca5},
+ {0x002ca7, 0x002ca7},
+ {0x002ca9, 0x002ca9},
+ {0x002cab, 0x002cab},
+ {0x002cad, 0x002cad},
+ {0x002caf, 0x002caf},
+ {0x002cb1, 0x002cb1},
+ {0x002cb3, 0x002cb3},
+ {0x002cb5, 0x002cb5},
+ {0x002cb7, 0x002cb7},
+ {0x002cb9, 0x002cb9},
+ {0x002cbb, 0x002cbb},
+ {0x002cbd, 0x002cbd},
+ {0x002cbf, 0x002cbf},
+ {0x002cc1, 0x002cc1},
+ {0x002cc3, 0x002cc3},
+ {0x002cc5, 0x002cc5},
+ {0x002cc7, 0x002cc7},
+ {0x002cc9, 0x002cc9},
+ {0x002ccb, 0x002ccb},
+ {0x002ccd, 0x002ccd},
+ {0x002ccf, 0x002ccf},
+ {0x002cd1, 0x002cd1},
+ {0x002cd3, 0x002cd3},
+ {0x002cd5, 0x002cd5},
+ {0x002cd7, 0x002cd7},
+ {0x002cd9, 0x002cd9},
+ {0x002cdb, 0x002cdb},
+ {0x002cdd, 0x002cdd},
+ {0x002cdf, 0x002cdf},
+ {0x002ce1, 0x002ce1},
+ {0x002ce3, 0x002ce4},
+ {0x002cec, 0x002cec},
+ {0x002cee, 0x002cee},
+ {0x002cf3, 0x002cf3},
+ {0x002d00, 0x002d25},
+ {0x002d27, 0x002d27},
+ {0x002d2d, 0x002d2d},
+ {0x00a641, 0x00a641},
+ {0x00a643, 0x00a643},
+ {0x00a645, 0x00a645},
+ {0x00a647, 0x00a647},
+ {0x00a649, 0x00a649},
+ {0x00a64b, 0x00a64b},
+ {0x00a64d, 0x00a64d},
+ {0x00a64f, 0x00a64f},
+ {0x00a651, 0x00a651},
+ {0x00a653, 0x00a653},
+ {0x00a655, 0x00a655},
+ {0x00a657, 0x00a657},
+ {0x00a659, 0x00a659},
+ {0x00a65b, 0x00a65b},
+ {0x00a65d, 0x00a65d},
+ {0x00a65f, 0x00a65f},
+ {0x00a661, 0x00a661},
+ {0x00a663, 0x00a663},
+ {0x00a665, 0x00a665},
+ {0x00a667, 0x00a667},
+ {0x00a669, 0x00a669},
+ {0x00a66b, 0x00a66b},
+ {0x00a66d, 0x00a66d},
+ {0x00a681, 0x00a681},
+ {0x00a683, 0x00a683},
+ {0x00a685, 0x00a685},
+ {0x00a687, 0x00a687},
+ {0x00a689, 0x00a689},
+ {0x00a68b, 0x00a68b},
+ {0x00a68d, 0x00a68d},
+ {0x00a68f, 0x00a68f},
+ {0x00a691, 0x00a691},
+ {0x00a693, 0x00a693},
+ {0x00a695, 0x00a695},
+ {0x00a697, 0x00a697},
+ {0x00a699, 0x00a699},
+ {0x00a69b, 0x00a69b},
+ {0x00a69c, 0x00a69d},
+ {0x00a723, 0x00a723},
+ {0x00a725, 0x00a725},
+ {0x00a727, 0x00a727},
+ {0x00a729, 0x00a729},
+ {0x00a72b, 0x00a72b},
+ {0x00a72d, 0x00a72d},
+ {0x00a72f, 0x00a731},
+ {0x00a733, 0x00a733},
+ {0x00a735, 0x00a735},
+ {0x00a737, 0x00a737},
+ {0x00a739, 0x00a739},
+ {0x00a73b, 0x00a73b},
+ {0x00a73d, 0x00a73d},
+ {0x00a73f, 0x00a73f},
+ {0x00a741, 0x00a741},
+ {0x00a743, 0x00a743},
+ {0x00a745, 0x00a745},
+ {0x00a747, 0x00a747},
+ {0x00a749, 0x00a749},
+ {0x00a74b, 0x00a74b},
+ {0x00a74d, 0x00a74d},
+ {0x00a74f, 0x00a74f},
+ {0x00a751, 0x00a751},
+ {0x00a753, 0x00a753},
+ {0x00a755, 0x00a755},
+ {0x00a757, 0x00a757},
+ {0x00a759, 0x00a759},
+ {0x00a75b, 0x00a75b},
+ {0x00a75d, 0x00a75d},
+ {0x00a75f, 0x00a75f},
+ {0x00a761, 0x00a761},
+ {0x00a763, 0x00a763},
+ {0x00a765, 0x00a765},
+ {0x00a767, 0x00a767},
+ {0x00a769, 0x00a769},
+ {0x00a76b, 0x00a76b},
+ {0x00a76d, 0x00a76d},
+ {0x00a76f, 0x00a76f},
+ {0x00a770, 0x00a770},
+ {0x00a771, 0x00a778},
+ {0x00a77a, 0x00a77a},
+ {0x00a77c, 0x00a77c},
+ {0x00a77f, 0x00a77f},
+ {0x00a781, 0x00a781},
+ {0x00a783, 0x00a783},
+ {0x00a785, 0x00a785},
+ {0x00a787, 0x00a787},
+ {0x00a78c, 0x00a78c},
+ {0x00a78e, 0x00a78e},
+ {0x00a791, 0x00a791},
+ {0x00a793, 0x00a795},
+ {0x00a797, 0x00a797},
+ {0x00a799, 0x00a799},
+ {0x00a79b, 0x00a79b},
+ {0x00a79d, 0x00a79d},
+ {0x00a79f, 0x00a79f},
+ {0x00a7a1, 0x00a7a1},
+ {0x00a7a3, 0x00a7a3},
+ {0x00a7a5, 0x00a7a5},
+ {0x00a7a7, 0x00a7a7},
+ {0x00a7a9, 0x00a7a9},
+ {0x00a7af, 0x00a7af},
+ {0x00a7b5, 0x00a7b5},
+ {0x00a7b7, 0x00a7b7},
+ {0x00a7b9, 0x00a7b9},
+ {0x00a7bb, 0x00a7bb},
+ {0x00a7bd, 0x00a7bd},
+ {0x00a7bf, 0x00a7bf},
+ {0x00a7c1, 0x00a7c1},
+ {0x00a7c3, 0x00a7c3},
+ {0x00a7c8, 0x00a7c8},
+ {0x00a7ca, 0x00a7ca},
+ {0x00a7d1, 0x00a7d1},
+ {0x00a7d3, 0x00a7d3},
+ {0x00a7d5, 0x00a7d5},
+ {0x00a7d7, 0x00a7d7},
+ {0x00a7d9, 0x00a7d9},
+ {0x00a7f2, 0x00a7f4},
+ {0x00a7f6, 0x00a7f6},
+ {0x00a7f8, 0x00a7f9},
+ {0x00a7fa, 0x00a7fa},
+ {0x00ab30, 0x00ab5a},
+ {0x00ab5c, 0x00ab5f},
+ {0x00ab60, 0x00ab68},
+ {0x00ab69, 0x00ab69},
+ {0x00ab70, 0x00abbf},
+ {0x00fb00, 0x00fb06},
+ {0x00fb13, 0x00fb17},
+ {0x00ff41, 0x00ff5a},
+ {0x010428, 0x01044f},
+ {0x0104d8, 0x0104fb},
+ {0x010597, 0x0105a1},
+ {0x0105a3, 0x0105b1},
+ {0x0105b3, 0x0105b9},
+ {0x0105bb, 0x0105bc},
+ {0x010780, 0x010780},
+ {0x010783, 0x010785},
+ {0x010787, 0x0107b0},
+ {0x0107b2, 0x0107ba},
+ {0x010cc0, 0x010cf2},
+ {0x0118c0, 0x0118df},
+ {0x016e60, 0x016e7f},
+ {0x01d41a, 0x01d433},
+ {0x01d44e, 0x01d454},
+ {0x01d456, 0x01d467},
+ {0x01d482, 0x01d49b},
+ {0x01d4b6, 0x01d4b9},
+ {0x01d4bb, 0x01d4bb},
+ {0x01d4bd, 0x01d4c3},
+ {0x01d4c5, 0x01d4cf},
+ {0x01d4ea, 0x01d503},
+ {0x01d51e, 0x01d537},
+ {0x01d552, 0x01d56b},
+ {0x01d586, 0x01d59f},
+ {0x01d5ba, 0x01d5d3},
+ {0x01d5ee, 0x01d607},
+ {0x01d622, 0x01d63b},
+ {0x01d656, 0x01d66f},
+ {0x01d68a, 0x01d6a5},
+ {0x01d6c2, 0x01d6da},
+ {0x01d6dc, 0x01d6e1},
+ {0x01d6fc, 0x01d714},
+ {0x01d716, 0x01d71b},
+ {0x01d736, 0x01d74e},
+ {0x01d750, 0x01d755},
+ {0x01d770, 0x01d788},
+ {0x01d78a, 0x01d78f},
+ {0x01d7aa, 0x01d7c2},
+ {0x01d7c4, 0x01d7c9},
+ {0x01d7cb, 0x01d7cb},
+ {0x01df00, 0x01df09},
+ {0x01df0b, 0x01df1e},
+ {0x01df25, 0x01df2a},
+ {0x01e030, 0x01e06d},
+ {0x01e922, 0x01e943}
+};
+
+/* table of Unicode codepoint ranges of Uppercase characters */
+static const pg_unicode_range unicode_uppercase[651] =
+{
+ {0x000041, 0x00005a},
+ {0x0000c0, 0x0000d6},
+ {0x0000d8, 0x0000de},
+ {0x000100, 0x000100},
+ {0x000102, 0x000102},
+ {0x000104, 0x000104},
+ {0x000106, 0x000106},
+ {0x000108, 0x000108},
+ {0x00010a, 0x00010a},
+ {0x00010c, 0x00010c},
+ {0x00010e, 0x00010e},
+ {0x000110, 0x000110},
+ {0x000112, 0x000112},
+ {0x000114, 0x000114},
+ {0x000116, 0x000116},
+ {0x000118, 0x000118},
+ {0x00011a, 0x00011a},
+ {0x00011c, 0x00011c},
+ {0x00011e, 0x00011e},
+ {0x000120, 0x000120},
+ {0x000122, 0x000122},
+ {0x000124, 0x000124},
+ {0x000126, 0x000126},
+ {0x000128, 0x000128},
+ {0x00012a, 0x00012a},
+ {0x00012c, 0x00012c},
+ {0x00012e, 0x00012e},
+ {0x000130, 0x000130},
+ {0x000132, 0x000132},
+ {0x000134, 0x000134},
+ {0x000136, 0x000136},
+ {0x000139, 0x000139},
+ {0x00013b, 0x00013b},
+ {0x00013d, 0x00013d},
+ {0x00013f, 0x00013f},
+ {0x000141, 0x000141},
+ {0x000143, 0x000143},
+ {0x000145, 0x000145},
+ {0x000147, 0x000147},
+ {0x00014a, 0x00014a},
+ {0x00014c, 0x00014c},
+ {0x00014e, 0x00014e},
+ {0x000150, 0x000150},
+ {0x000152, 0x000152},
+ {0x000154, 0x000154},
+ {0x000156, 0x000156},
+ {0x000158, 0x000158},
+ {0x00015a, 0x00015a},
+ {0x00015c, 0x00015c},
+ {0x00015e, 0x00015e},
+ {0x000160, 0x000160},
+ {0x000162, 0x000162},
+ {0x000164, 0x000164},
+ {0x000166, 0x000166},
+ {0x000168, 0x000168},
+ {0x00016a, 0x00016a},
+ {0x00016c, 0x00016c},
+ {0x00016e, 0x00016e},
+ {0x000170, 0x000170},
+ {0x000172, 0x000172},
+ {0x000174, 0x000174},
+ {0x000176, 0x000176},
+ {0x000178, 0x000179},
+ {0x00017b, 0x00017b},
+ {0x00017d, 0x00017d},
+ {0x000181, 0x000182},
+ {0x000184, 0x000184},
+ {0x000186, 0x000187},
+ {0x000189, 0x00018b},
+ {0x00018e, 0x000191},
+ {0x000193, 0x000194},
+ {0x000196, 0x000198},
+ {0x00019c, 0x00019d},
+ {0x00019f, 0x0001a0},
+ {0x0001a2, 0x0001a2},
+ {0x0001a4, 0x0001a4},
+ {0x0001a6, 0x0001a7},
+ {0x0001a9, 0x0001a9},
+ {0x0001ac, 0x0001ac},
+ {0x0001ae, 0x0001af},
+ {0x0001b1, 0x0001b3},
+ {0x0001b5, 0x0001b5},
+ {0x0001b7, 0x0001b8},
+ {0x0001bc, 0x0001bc},
+ {0x0001c4, 0x0001c4},
+ {0x0001c7, 0x0001c7},
+ {0x0001ca, 0x0001ca},
+ {0x0001cd, 0x0001cd},
+ {0x0001cf, 0x0001cf},
+ {0x0001d1, 0x0001d1},
+ {0x0001d3, 0x0001d3},
+ {0x0001d5, 0x0001d5},
+ {0x0001d7, 0x0001d7},
+ {0x0001d9, 0x0001d9},
+ {0x0001db, 0x0001db},
+ {0x0001de, 0x0001de},
+ {0x0001e0, 0x0001e0},
+ {0x0001e2, 0x0001e2},
+ {0x0001e4, 0x0001e4},
+ {0x0001e6, 0x0001e6},
+ {0x0001e8, 0x0001e8},
+ {0x0001ea, 0x0001ea},
+ {0x0001ec, 0x0001ec},
+ {0x0001ee, 0x0001ee},
+ {0x0001f1, 0x0001f1},
+ {0x0001f4, 0x0001f4},
+ {0x0001f6, 0x0001f8},
+ {0x0001fa, 0x0001fa},
+ {0x0001fc, 0x0001fc},
+ {0x0001fe, 0x0001fe},
+ {0x000200, 0x000200},
+ {0x000202, 0x000202},
+ {0x000204, 0x000204},
+ {0x000206, 0x000206},
+ {0x000208, 0x000208},
+ {0x00020a, 0x00020a},
+ {0x00020c, 0x00020c},
+ {0x00020e, 0x00020e},
+ {0x000210, 0x000210},
+ {0x000212, 0x000212},
+ {0x000214, 0x000214},
+ {0x000216, 0x000216},
+ {0x000218, 0x000218},
+ {0x00021a, 0x00021a},
+ {0x00021c, 0x00021c},
+ {0x00021e, 0x00021e},
+ {0x000220, 0x000220},
+ {0x000222, 0x000222},
+ {0x000224, 0x000224},
+ {0x000226, 0x000226},
+ {0x000228, 0x000228},
+ {0x00022a, 0x00022a},
+ {0x00022c, 0x00022c},
+ {0x00022e, 0x00022e},
+ {0x000230, 0x000230},
+ {0x000232, 0x000232},
+ {0x00023a, 0x00023b},
+ {0x00023d, 0x00023e},
+ {0x000241, 0x000241},
+ {0x000243, 0x000246},
+ {0x000248, 0x000248},
+ {0x00024a, 0x00024a},
+ {0x00024c, 0x00024c},
+ {0x00024e, 0x00024e},
+ {0x000370, 0x000370},
+ {0x000372, 0x000372},
+ {0x000376, 0x000376},
+ {0x00037f, 0x00037f},
+ {0x000386, 0x000386},
+ {0x000388, 0x00038a},
+ {0x00038c, 0x00038c},
+ {0x00038e, 0x00038f},
+ {0x000391, 0x0003a1},
+ {0x0003a3, 0x0003ab},
+ {0x0003cf, 0x0003cf},
+ {0x0003d2, 0x0003d4},
+ {0x0003d8, 0x0003d8},
+ {0x0003da, 0x0003da},
+ {0x0003dc, 0x0003dc},
+ {0x0003de, 0x0003de},
+ {0x0003e0, 0x0003e0},
+ {0x0003e2, 0x0003e2},
+ {0x0003e4, 0x0003e4},
+ {0x0003e6, 0x0003e6},
+ {0x0003e8, 0x0003e8},
+ {0x0003ea, 0x0003ea},
+ {0x0003ec, 0x0003ec},
+ {0x0003ee, 0x0003ee},
+ {0x0003f4, 0x0003f4},
+ {0x0003f7, 0x0003f7},
+ {0x0003f9, 0x0003fa},
+ {0x0003fd, 0x00042f},
+ {0x000460, 0x000460},
+ {0x000462, 0x000462},
+ {0x000464, 0x000464},
+ {0x000466, 0x000466},
+ {0x000468, 0x000468},
+ {0x00046a, 0x00046a},
+ {0x00046c, 0x00046c},
+ {0x00046e, 0x00046e},
+ {0x000470, 0x000470},
+ {0x000472, 0x000472},
+ {0x000474, 0x000474},
+ {0x000476, 0x000476},
+ {0x000478, 0x000478},
+ {0x00047a, 0x00047a},
+ {0x00047c, 0x00047c},
+ {0x00047e, 0x00047e},
+ {0x000480, 0x000480},
+ {0x00048a, 0x00048a},
+ {0x00048c, 0x00048c},
+ {0x00048e, 0x00048e},
+ {0x000490, 0x000490},
+ {0x000492, 0x000492},
+ {0x000494, 0x000494},
+ {0x000496, 0x000496},
+ {0x000498, 0x000498},
+ {0x00049a, 0x00049a},
+ {0x00049c, 0x00049c},
+ {0x00049e, 0x00049e},
+ {0x0004a0, 0x0004a0},
+ {0x0004a2, 0x0004a2},
+ {0x0004a4, 0x0004a4},
+ {0x0004a6, 0x0004a6},
+ {0x0004a8, 0x0004a8},
+ {0x0004aa, 0x0004aa},
+ {0x0004ac, 0x0004ac},
+ {0x0004ae, 0x0004ae},
+ {0x0004b0, 0x0004b0},
+ {0x0004b2, 0x0004b2},
+ {0x0004b4, 0x0004b4},
+ {0x0004b6, 0x0004b6},
+ {0x0004b8, 0x0004b8},
+ {0x0004ba, 0x0004ba},
+ {0x0004bc, 0x0004bc},
+ {0x0004be, 0x0004be},
+ {0x0004c0, 0x0004c1},
+ {0x0004c3, 0x0004c3},
+ {0x0004c5, 0x0004c5},
+ {0x0004c7, 0x0004c7},
+ {0x0004c9, 0x0004c9},
+ {0x0004cb, 0x0004cb},
+ {0x0004cd, 0x0004cd},
+ {0x0004d0, 0x0004d0},
+ {0x0004d2, 0x0004d2},
+ {0x0004d4, 0x0004d4},
+ {0x0004d6, 0x0004d6},
+ {0x0004d8, 0x0004d8},
+ {0x0004da, 0x0004da},
+ {0x0004dc, 0x0004dc},
+ {0x0004de, 0x0004de},
+ {0x0004e0, 0x0004e0},
+ {0x0004e2, 0x0004e2},
+ {0x0004e4, 0x0004e4},
+ {0x0004e6, 0x0004e6},
+ {0x0004e8, 0x0004e8},
+ {0x0004ea, 0x0004ea},
+ {0x0004ec, 0x0004ec},
+ {0x0004ee, 0x0004ee},
+ {0x0004f0, 0x0004f0},
+ {0x0004f2, 0x0004f2},
+ {0x0004f4, 0x0004f4},
+ {0x0004f6, 0x0004f6},
+ {0x0004f8, 0x0004f8},
+ {0x0004fa, 0x0004fa},
+ {0x0004fc, 0x0004fc},
+ {0x0004fe, 0x0004fe},
+ {0x000500, 0x000500},
+ {0x000502, 0x000502},
+ {0x000504, 0x000504},
+ {0x000506, 0x000506},
+ {0x000508, 0x000508},
+ {0x00050a, 0x00050a},
+ {0x00050c, 0x00050c},
+ {0x00050e, 0x00050e},
+ {0x000510, 0x000510},
+ {0x000512, 0x000512},
+ {0x000514, 0x000514},
+ {0x000516, 0x000516},
+ {0x000518, 0x000518},
+ {0x00051a, 0x00051a},
+ {0x00051c, 0x00051c},
+ {0x00051e, 0x00051e},
+ {0x000520, 0x000520},
+ {0x000522, 0x000522},
+ {0x000524, 0x000524},
+ {0x000526, 0x000526},
+ {0x000528, 0x000528},
+ {0x00052a, 0x00052a},
+ {0x00052c, 0x00052c},
+ {0x00052e, 0x00052e},
+ {0x000531, 0x000556},
+ {0x0010a0, 0x0010c5},
+ {0x0010c7, 0x0010c7},
+ {0x0010cd, 0x0010cd},
+ {0x0013a0, 0x0013f5},
+ {0x001c90, 0x001cba},
+ {0x001cbd, 0x001cbf},
+ {0x001e00, 0x001e00},
+ {0x001e02, 0x001e02},
+ {0x001e04, 0x001e04},
+ {0x001e06, 0x001e06},
+ {0x001e08, 0x001e08},
+ {0x001e0a, 0x001e0a},
+ {0x001e0c, 0x001e0c},
+ {0x001e0e, 0x001e0e},
+ {0x001e10, 0x001e10},
+ {0x001e12, 0x001e12},
+ {0x001e14, 0x001e14},
+ {0x001e16, 0x001e16},
+ {0x001e18, 0x001e18},
+ {0x001e1a, 0x001e1a},
+ {0x001e1c, 0x001e1c},
+ {0x001e1e, 0x001e1e},
+ {0x001e20, 0x001e20},
+ {0x001e22, 0x001e22},
+ {0x001e24, 0x001e24},
+ {0x001e26, 0x001e26},
+ {0x001e28, 0x001e28},
+ {0x001e2a, 0x001e2a},
+ {0x001e2c, 0x001e2c},
+ {0x001e2e, 0x001e2e},
+ {0x001e30, 0x001e30},
+ {0x001e32, 0x001e32},
+ {0x001e34, 0x001e34},
+ {0x001e36, 0x001e36},
+ {0x001e38, 0x001e38},
+ {0x001e3a, 0x001e3a},
+ {0x001e3c, 0x001e3c},
+ {0x001e3e, 0x001e3e},
+ {0x001e40, 0x001e40},
+ {0x001e42, 0x001e42},
+ {0x001e44, 0x001e44},
+ {0x001e46, 0x001e46},
+ {0x001e48, 0x001e48},
+ {0x001e4a, 0x001e4a},
+ {0x001e4c, 0x001e4c},
+ {0x001e4e, 0x001e4e},
+ {0x001e50, 0x001e50},
+ {0x001e52, 0x001e52},
+ {0x001e54, 0x001e54},
+ {0x001e56, 0x001e56},
+ {0x001e58, 0x001e58},
+ {0x001e5a, 0x001e5a},
+ {0x001e5c, 0x001e5c},
+ {0x001e5e, 0x001e5e},
+ {0x001e60, 0x001e60},
+ {0x001e62, 0x001e62},
+ {0x001e64, 0x001e64},
+ {0x001e66, 0x001e66},
+ {0x001e68, 0x001e68},
+ {0x001e6a, 0x001e6a},
+ {0x001e6c, 0x001e6c},
+ {0x001e6e, 0x001e6e},
+ {0x001e70, 0x001e70},
+ {0x001e72, 0x001e72},
+ {0x001e74, 0x001e74},
+ {0x001e76, 0x001e76},
+ {0x001e78, 0x001e78},
+ {0x001e7a, 0x001e7a},
+ {0x001e7c, 0x001e7c},
+ {0x001e7e, 0x001e7e},
+ {0x001e80, 0x001e80},
+ {0x001e82, 0x001e82},
+ {0x001e84, 0x001e84},
+ {0x001e86, 0x001e86},
+ {0x001e88, 0x001e88},
+ {0x001e8a, 0x001e8a},
+ {0x001e8c, 0x001e8c},
+ {0x001e8e, 0x001e8e},
+ {0x001e90, 0x001e90},
+ {0x001e92, 0x001e92},
+ {0x001e94, 0x001e94},
+ {0x001e9e, 0x001e9e},
+ {0x001ea0, 0x001ea0},
+ {0x001ea2, 0x001ea2},
+ {0x001ea4, 0x001ea4},
+ {0x001ea6, 0x001ea6},
+ {0x001ea8, 0x001ea8},
+ {0x001eaa, 0x001eaa},
+ {0x001eac, 0x001eac},
+ {0x001eae, 0x001eae},
+ {0x001eb0, 0x001eb0},
+ {0x001eb2, 0x001eb2},
+ {0x001eb4, 0x001eb4},
+ {0x001eb6, 0x001eb6},
+ {0x001eb8, 0x001eb8},
+ {0x001eba, 0x001eba},
+ {0x001ebc, 0x001ebc},
+ {0x001ebe, 0x001ebe},
+ {0x001ec0, 0x001ec0},
+ {0x001ec2, 0x001ec2},
+ {0x001ec4, 0x001ec4},
+ {0x001ec6, 0x001ec6},
+ {0x001ec8, 0x001ec8},
+ {0x001eca, 0x001eca},
+ {0x001ecc, 0x001ecc},
+ {0x001ece, 0x001ece},
+ {0x001ed0, 0x001ed0},
+ {0x001ed2, 0x001ed2},
+ {0x001ed4, 0x001ed4},
+ {0x001ed6, 0x001ed6},
+ {0x001ed8, 0x001ed8},
+ {0x001eda, 0x001eda},
+ {0x001edc, 0x001edc},
+ {0x001ede, 0x001ede},
+ {0x001ee0, 0x001ee0},
+ {0x001ee2, 0x001ee2},
+ {0x001ee4, 0x001ee4},
+ {0x001ee6, 0x001ee6},
+ {0x001ee8, 0x001ee8},
+ {0x001eea, 0x001eea},
+ {0x001eec, 0x001eec},
+ {0x001eee, 0x001eee},
+ {0x001ef0, 0x001ef0},
+ {0x001ef2, 0x001ef2},
+ {0x001ef4, 0x001ef4},
+ {0x001ef6, 0x001ef6},
+ {0x001ef8, 0x001ef8},
+ {0x001efa, 0x001efa},
+ {0x001efc, 0x001efc},
+ {0x001efe, 0x001efe},
+ {0x001f08, 0x001f0f},
+ {0x001f18, 0x001f1d},
+ {0x001f28, 0x001f2f},
+ {0x001f38, 0x001f3f},
+ {0x001f48, 0x001f4d},
+ {0x001f59, 0x001f59},
+ {0x001f5b, 0x001f5b},
+ {0x001f5d, 0x001f5d},
+ {0x001f5f, 0x001f5f},
+ {0x001f68, 0x001f6f},
+ {0x001fb8, 0x001fbb},
+ {0x001fc8, 0x001fcb},
+ {0x001fd8, 0x001fdb},
+ {0x001fe8, 0x001fec},
+ {0x001ff8, 0x001ffb},
+ {0x002102, 0x002102},
+ {0x002107, 0x002107},
+ {0x00210b, 0x00210d},
+ {0x002110, 0x002112},
+ {0x002115, 0x002115},
+ {0x002119, 0x00211d},
+ {0x002124, 0x002124},
+ {0x002126, 0x002126},
+ {0x002128, 0x002128},
+ {0x00212a, 0x00212d},
+ {0x002130, 0x002133},
+ {0x00213e, 0x00213f},
+ {0x002145, 0x002145},
+ {0x002160, 0x00216f},
+ {0x002183, 0x002183},
+ {0x0024b6, 0x0024cf},
+ {0x002c00, 0x002c2f},
+ {0x002c60, 0x002c60},
+ {0x002c62, 0x002c64},
+ {0x002c67, 0x002c67},
+ {0x002c69, 0x002c69},
+ {0x002c6b, 0x002c6b},
+ {0x002c6d, 0x002c70},
+ {0x002c72, 0x002c72},
+ {0x002c75, 0x002c75},
+ {0x002c7e, 0x002c80},
+ {0x002c82, 0x002c82},
+ {0x002c84, 0x002c84},
+ {0x002c86, 0x002c86},
+ {0x002c88, 0x002c88},
+ {0x002c8a, 0x002c8a},
+ {0x002c8c, 0x002c8c},
+ {0x002c8e, 0x002c8e},
+ {0x002c90, 0x002c90},
+ {0x002c92, 0x002c92},
+ {0x002c94, 0x002c94},
+ {0x002c96, 0x002c96},
+ {0x002c98, 0x002c98},
+ {0x002c9a, 0x002c9a},
+ {0x002c9c, 0x002c9c},
+ {0x002c9e, 0x002c9e},
+ {0x002ca0, 0x002ca0},
+ {0x002ca2, 0x002ca2},
+ {0x002ca4, 0x002ca4},
+ {0x002ca6, 0x002ca6},
+ {0x002ca8, 0x002ca8},
+ {0x002caa, 0x002caa},
+ {0x002cac, 0x002cac},
+ {0x002cae, 0x002cae},
+ {0x002cb0, 0x002cb0},
+ {0x002cb2, 0x002cb2},
+ {0x002cb4, 0x002cb4},
+ {0x002cb6, 0x002cb6},
+ {0x002cb8, 0x002cb8},
+ {0x002cba, 0x002cba},
+ {0x002cbc, 0x002cbc},
+ {0x002cbe, 0x002cbe},
+ {0x002cc0, 0x002cc0},
+ {0x002cc2, 0x002cc2},
+ {0x002cc4, 0x002cc4},
+ {0x002cc6, 0x002cc6},
+ {0x002cc8, 0x002cc8},
+ {0x002cca, 0x002cca},
+ {0x002ccc, 0x002ccc},
+ {0x002cce, 0x002cce},
+ {0x002cd0, 0x002cd0},
+ {0x002cd2, 0x002cd2},
+ {0x002cd4, 0x002cd4},
+ {0x002cd6, 0x002cd6},
+ {0x002cd8, 0x002cd8},
+ {0x002cda, 0x002cda},
+ {0x002cdc, 0x002cdc},
+ {0x002cde, 0x002cde},
+ {0x002ce0, 0x002ce0},
+ {0x002ce2, 0x002ce2},
+ {0x002ceb, 0x002ceb},
+ {0x002ced, 0x002ced},
+ {0x002cf2, 0x002cf2},
+ {0x00a640, 0x00a640},
+ {0x00a642, 0x00a642},
+ {0x00a644, 0x00a644},
+ {0x00a646, 0x00a646},
+ {0x00a648, 0x00a648},
+ {0x00a64a, 0x00a64a},
+ {0x00a64c, 0x00a64c},
+ {0x00a64e, 0x00a64e},
+ {0x00a650, 0x00a650},
+ {0x00a652, 0x00a652},
+ {0x00a654, 0x00a654},
+ {0x00a656, 0x00a656},
+ {0x00a658, 0x00a658},
+ {0x00a65a, 0x00a65a},
+ {0x00a65c, 0x00a65c},
+ {0x00a65e, 0x00a65e},
+ {0x00a660, 0x00a660},
+ {0x00a662, 0x00a662},
+ {0x00a664, 0x00a664},
+ {0x00a666, 0x00a666},
+ {0x00a668, 0x00a668},
+ {0x00a66a, 0x00a66a},
+ {0x00a66c, 0x00a66c},
+ {0x00a680, 0x00a680},
+ {0x00a682, 0x00a682},
+ {0x00a684, 0x00a684},
+ {0x00a686, 0x00a686},
+ {0x00a688, 0x00a688},
+ {0x00a68a, 0x00a68a},
+ {0x00a68c, 0x00a68c},
+ {0x00a68e, 0x00a68e},
+ {0x00a690, 0x00a690},
+ {0x00a692, 0x00a692},
+ {0x00a694, 0x00a694},
+ {0x00a696, 0x00a696},
+ {0x00a698, 0x00a698},
+ {0x00a69a, 0x00a69a},
+ {0x00a722, 0x00a722},
+ {0x00a724, 0x00a724},
+ {0x00a726, 0x00a726},
+ {0x00a728, 0x00a728},
+ {0x00a72a, 0x00a72a},
+ {0x00a72c, 0x00a72c},
+ {0x00a72e, 0x00a72e},
+ {0x00a732, 0x00a732},
+ {0x00a734, 0x00a734},
+ {0x00a736, 0x00a736},
+ {0x00a738, 0x00a738},
+ {0x00a73a, 0x00a73a},
+ {0x00a73c, 0x00a73c},
+ {0x00a73e, 0x00a73e},
+ {0x00a740, 0x00a740},
+ {0x00a742, 0x00a742},
+ {0x00a744, 0x00a744},
+ {0x00a746, 0x00a746},
+ {0x00a748, 0x00a748},
+ {0x00a74a, 0x00a74a},
+ {0x00a74c, 0x00a74c},
+ {0x00a74e, 0x00a74e},
+ {0x00a750, 0x00a750},
+ {0x00a752, 0x00a752},
+ {0x00a754, 0x00a754},
+ {0x00a756, 0x00a756},
+ {0x00a758, 0x00a758},
+ {0x00a75a, 0x00a75a},
+ {0x00a75c, 0x00a75c},
+ {0x00a75e, 0x00a75e},
+ {0x00a760, 0x00a760},
+ {0x00a762, 0x00a762},
+ {0x00a764, 0x00a764},
+ {0x00a766, 0x00a766},
+ {0x00a768, 0x00a768},
+ {0x00a76a, 0x00a76a},
+ {0x00a76c, 0x00a76c},
+ {0x00a76e, 0x00a76e},
+ {0x00a779, 0x00a779},
+ {0x00a77b, 0x00a77b},
+ {0x00a77d, 0x00a77e},
+ {0x00a780, 0x00a780},
+ {0x00a782, 0x00a782},
+ {0x00a784, 0x00a784},
+ {0x00a786, 0x00a786},
+ {0x00a78b, 0x00a78b},
+ {0x00a78d, 0x00a78d},
+ {0x00a790, 0x00a790},
+ {0x00a792, 0x00a792},
+ {0x00a796, 0x00a796},
+ {0x00a798, 0x00a798},
+ {0x00a79a, 0x00a79a},
+ {0x00a79c, 0x00a79c},
+ {0x00a79e, 0x00a79e},
+ {0x00a7a0, 0x00a7a0},
+ {0x00a7a2, 0x00a7a2},
+ {0x00a7a4, 0x00a7a4},
+ {0x00a7a6, 0x00a7a6},
+ {0x00a7a8, 0x00a7a8},
+ {0x00a7aa, 0x00a7ae},
+ {0x00a7b0, 0x00a7b4},
+ {0x00a7b6, 0x00a7b6},
+ {0x00a7b8, 0x00a7b8},
+ {0x00a7ba, 0x00a7ba},
+ {0x00a7bc, 0x00a7bc},
+ {0x00a7be, 0x00a7be},
+ {0x00a7c0, 0x00a7c0},
+ {0x00a7c2, 0x00a7c2},
+ {0x00a7c4, 0x00a7c7},
+ {0x00a7c9, 0x00a7c9},
+ {0x00a7d0, 0x00a7d0},
+ {0x00a7d6, 0x00a7d6},
+ {0x00a7d8, 0x00a7d8},
+ {0x00a7f5, 0x00a7f5},
+ {0x00ff21, 0x00ff3a},
+ {0x010400, 0x010427},
+ {0x0104b0, 0x0104d3},
+ {0x010570, 0x01057a},
+ {0x01057c, 0x01058a},
+ {0x01058c, 0x010592},
+ {0x010594, 0x010595},
+ {0x010c80, 0x010cb2},
+ {0x0118a0, 0x0118bf},
+ {0x016e40, 0x016e5f},
+ {0x01d400, 0x01d419},
+ {0x01d434, 0x01d44d},
+ {0x01d468, 0x01d481},
+ {0x01d49c, 0x01d49c},
+ {0x01d49e, 0x01d49f},
+ {0x01d4a2, 0x01d4a2},
+ {0x01d4a5, 0x01d4a6},
+ {0x01d4a9, 0x01d4ac},
+ {0x01d4ae, 0x01d4b5},
+ {0x01d4d0, 0x01d4e9},
+ {0x01d504, 0x01d505},
+ {0x01d507, 0x01d50a},
+ {0x01d50d, 0x01d514},
+ {0x01d516, 0x01d51c},
+ {0x01d538, 0x01d539},
+ {0x01d53b, 0x01d53e},
+ {0x01d540, 0x01d544},
+ {0x01d546, 0x01d546},
+ {0x01d54a, 0x01d550},
+ {0x01d56c, 0x01d585},
+ {0x01d5a0, 0x01d5b9},
+ {0x01d5d4, 0x01d5ed},
+ {0x01d608, 0x01d621},
+ {0x01d63c, 0x01d655},
+ {0x01d670, 0x01d689},
+ {0x01d6a8, 0x01d6c0},
+ {0x01d6e2, 0x01d6fa},
+ {0x01d71c, 0x01d734},
+ {0x01d756, 0x01d76e},
+ {0x01d790, 0x01d7a8},
+ {0x01d7ca, 0x01d7ca},
+ {0x01e900, 0x01e921},
+ {0x01f130, 0x01f149},
+ {0x01f150, 0x01f169},
+ {0x01f170, 0x01f189}
+};
+
+/* table of Unicode codepoint ranges of White_Space characters */
+static const pg_unicode_range unicode_white_space[11] =
+{
+ {0x000009, 0x00000d},
+ {0x000020, 0x000020},
+ {0x000085, 0x000085},
+ {0x0000a0, 0x0000a0},
+ {0x001680, 0x001680},
+ {0x002000, 0x00200a},
+ {0x002028, 0x002028},
+ {0x002029, 0x002029},
+ {0x00202f, 0x00202f},
+ {0x00205f, 0x00205f},
+ {0x003000, 0x003000}
+};
+
+/* table of Unicode codepoint ranges of Hex_Digit characters */
+static const pg_unicode_range unicode_hex_digit[6] =
+{
+ {0x000030, 0x000039},
+ {0x000041, 0x000046},
+ {0x000061, 0x000066},
+ {0x00ff10, 0x00ff19},
+ {0x00ff21, 0x00ff26},
+ {0x00ff41, 0x00ff46}
+};
+
+/* table of Unicode codepoint ranges of Join_Control characters */
+static const pg_unicode_range unicode_join_control[1] =
+{
+ {0x00200c, 0x00200d}
+};
--
2.34.1
[text/x-patch] v15-0003-Add-unicode-case-mapping-tables-and-functions.patch (144.8K, ../../[email protected]/4-v15-0003-Add-unicode-case-mapping-tables-and-functions.patch)
download | inline diff:
From 53258636a18990937d7c1ba7e60ec645762d4fab Mon Sep 17 00:00:00 2001
From: Jeff Davis <[email protected]>
Date: Mon, 30 Oct 2023 17:38:54 -0700
Subject: [PATCH v15 3/5] Add unicode case mapping tables and functions.
---
src/common/Makefile | 1 +
src/common/meson.build | 1 +
src/common/unicode/Makefile | 13 +-
src/common/unicode/case_test.c | 128 +
.../unicode/generate-unicode_case_table.pl | 134 +
src/common/unicode/meson.build | 31 +
src/common/unicode_case.c | 87 +
src/include/common/unicode_case.h | 23 +
src/include/common/unicode_case_table.h | 2994 +++++++++++++++++
9 files changed, 3410 insertions(+), 2 deletions(-)
create mode 100644 src/common/unicode/case_test.c
create mode 100644 src/common/unicode/generate-unicode_case_table.pl
create mode 100644 src/common/unicode_case.c
create mode 100644 src/include/common/unicode_case.h
create mode 100644 src/include/common/unicode_case_table.h
diff --git a/src/common/Makefile b/src/common/Makefile
index 2ba5069dca..3d83299432 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -78,6 +78,7 @@ OBJS_COMMON = \
scram-common.o \
string.o \
stringinfo.o \
+ unicode_case.o \
unicode_category.o \
unicode_norm.o \
username.o \
diff --git a/src/common/meson.build b/src/common/meson.build
index 12fd43e87f..94c3e9529e 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -32,6 +32,7 @@ common_sources = files(
'scram-common.c',
'string.c',
'stringinfo.c',
+ 'unicode_case.c',
'unicode_category.c',
'unicode_norm.c',
'username.c',
diff --git a/src/common/unicode/Makefile b/src/common/unicode/Makefile
index 27f0408d8b..8223d02375 100644
--- a/src/common/unicode/Makefile
+++ b/src/common/unicode/Makefile
@@ -21,8 +21,9 @@ CPPFLAGS += $(ICU_CFLAGS)
# By default, do nothing.
all:
-update-unicode: unicode_category_table.h unicode_east_asian_fw_table.h unicode_nonspacing_table.h unicode_norm_hashfunc.h unicode_norm_table.h unicode_normprops_table.h unicode_version.h
+update-unicode: unicode_case_table.h unicode_category_table.h unicode_east_asian_fw_table.h unicode_nonspacing_table.h unicode_norm_hashfunc.h unicode_norm_table.h unicode_normprops_table.h unicode_version.h
mv $^ $(top_srcdir)/src/include/common/
+ $(MAKE) case-check
$(MAKE) category-check
$(MAKE) normalization-check
@@ -35,6 +36,9 @@ CompositionExclusions.txt DerivedCoreProperties.txt DerivedNormalizationProps.tx
unicode_version.h: generate-unicode_version.pl
$(PERL) $< --version $(UNICODE_VERSION)
+unicode_case_table.h: generate-unicode_case_table.pl UnicodeData.txt
+ $(PERL) $<
+
unicode_category_table.h: generate-unicode_category_table.pl DerivedCoreProperties.txt PropList.txt UnicodeData.txt
$(PERL) $<
@@ -55,12 +59,17 @@ unicode_normprops_table.h: generate-unicode_normprops_table.pl DerivedNormalizat
$(PERL) $^ >$@
# Test suite
+case-check: case_test
+ ./case_test
+
category-check: category_test
./category_test
normalization-check: norm_test
./norm_test
+case_test: case_test.o ../unicode_case.o | submake-common
+
category_test: category_test.o ../unicode_category.o | submake-common
norm_test: norm_test.o ../unicode_norm.o | submake-common
@@ -82,4 +91,4 @@ clean:
rm -f $(OBJS) category_test category_test.o norm_test norm_test.o
distclean: clean
- rm -f CompositionExclusions.txt DerivedCoreProperties.txt DerivedNormalizationProps.txt EastAsianWidth.txt NormalizationTest.txt PropList.txt UnicodeData.txt norm_test_table.h unicode_category_table.h unicode_norm_table.h
+ rm -f CompositionExclusions.txt DerivedCoreProperties.txt DerivedNormalizationProps.txt EastAsianWidth.txt NormalizationTest.txt PropList.txt UnicodeData.txt norm_test_table.h unicode_case_table.h unicode_category_table.h unicode_norm_table.h
diff --git a/src/common/unicode/case_test.c b/src/common/unicode/case_test.c
new file mode 100644
index 0000000000..6aae12a2f6
--- /dev/null
+++ b/src/common/unicode/case_test.c
@@ -0,0 +1,128 @@
+/*-------------------------------------------------------------------------
+ * case_test.c
+ * Program to test Unicode case mapping functions.
+ *
+ * Portions Copyright (c) 2017-2023, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/common/unicode/case_test.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres_fe.h"
+
+#include <locale.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <wctype.h>
+
+#ifdef USE_ICU
+#include <unicode/uchar.h>
+#endif
+#include "common/unicode_case.h"
+#include "common/unicode_category.h"
+#include "common/unicode_version.h"
+
+/*
+ * We expect that C.UTF-8 has the same CTYPE behavior as the simple unicode
+ * mappings, but that's not guaranteed. If there are failures in the libc
+ * test, that's useful information, but does not necessarily indicate a
+ * problem.
+ */
+#define LIBC_LOCALE "C.UTF-8"
+
+#ifdef USE_ICU
+
+/* use root locale for test */
+#define ICU_LOCALE "und"
+
+static void
+icu_test_simple(pg_wchar code)
+{
+ pg_wchar lower = unicode_lowercase_simple(code);
+ pg_wchar title = unicode_titlecase_simple(code);
+ pg_wchar upper = unicode_uppercase_simple(code);
+ pg_wchar iculower = u_tolower(code);
+ pg_wchar icutitle = u_totitle(code);
+ pg_wchar icuupper = u_toupper(code);
+
+ if (lower != iculower || title != icutitle || upper != icuupper)
+ {
+ printf("case_test: FAILURE for codepoint 0x%06x\n", code);
+ printf("case_test: Postgres lower/title/upper: 0x%06x/0x%06x/0x%06x\n",
+ lower, title, upper);
+ printf("case_test: ICU lower/title/upper: 0x%06x/0x%06x/0x%06x\n",
+ iculower, icutitle, icuupper);
+ printf("\n");
+ exit(1);
+ }
+}
+#endif
+
+static void
+libc_test_simple(pg_wchar code)
+{
+ pg_wchar lower = unicode_lowercase_simple(code);
+ pg_wchar upper = unicode_uppercase_simple(code);
+ wchar_t libclower = towlower(code);
+ wchar_t libcupper = towupper(code);
+
+ if (lower != libclower || upper != libcupper)
+ {
+ printf("case_test: FAILURE for codepoint 0x%06x\n", code);
+ printf("case_test: Postgres lower/upper: 0x%06x/0x%06x\n",
+ lower, upper);
+ printf("case_test: libc lower/upper: 0x%06x/0x%06x\n",
+ libclower, libcupper);
+ printf("\n");
+ exit(1);
+ }
+}
+
+/*
+ * Exhaustively compare case mappings with the results from libc and ICU.
+ */
+int
+main(int argc, char **argv)
+{
+ char *libc_locale;
+
+ libc_locale = setlocale(LC_CTYPE, LIBC_LOCALE);
+
+ printf("case_test: Postgres Unicode version:\t%s\n", PG_UNICODE_VERSION);
+#ifdef USE_ICU
+ printf("case_test: ICU Unicode version:\t\t%s\n", U_UNICODE_VERSION);
+#else
+ printf("case_test: ICU not available; skipping\n");
+#endif
+
+ if (libc_locale)
+ {
+ printf("case_test: comparing with libc locale \"%s\"\n", libc_locale);
+ for (pg_wchar code = 0; code <= 0x10ffff; code++)
+ {
+ pg_unicode_category category = unicode_category(code);
+
+ if (category != PG_U_UNASSIGNED && category != PG_U_SURROGATE)
+ libc_test_simple(code);
+ }
+ printf("case_test: libc simple mapping test successful\n");
+ }
+ else
+ printf("case_test: libc locale \"%s\" not available; skipping\n",
+ LIBC_LOCALE);
+
+#ifdef USE_ICU
+ for (pg_wchar code = 0; code <= 0x10ffff; code++)
+ {
+ pg_unicode_category category = unicode_category(code);
+
+ if (category != PG_U_UNASSIGNED && category != PG_U_SURROGATE)
+ icu_test_simple(code);
+ }
+ printf("case_test: ICU simple mapping test successful\n");
+#endif
+
+ exit(0);
+}
diff --git a/src/common/unicode/generate-unicode_case_table.pl b/src/common/unicode/generate-unicode_case_table.pl
new file mode 100644
index 0000000000..9bf1deb297
--- /dev/null
+++ b/src/common/unicode/generate-unicode_case_table.pl
@@ -0,0 +1,134 @@
+#!/usr/bin/perl
+#
+# Generate Unicode character case mappings. Does not include tailoring
+# or locale-specific mappings.
+#
+# Input: UnicodeData.txt
+# Output: unicode_case_table.h
+#
+# Copyright (c) 2000-2023, PostgreSQL Global Development Group
+
+use strict;
+use warnings;
+use Getopt::Long;
+
+use FindBin;
+use lib "$FindBin::RealBin/../../tools/";
+
+my $output_path = '.';
+
+GetOptions('outdir:s' => \$output_path);
+
+my $output_table_file = "$output_path/unicode_case_table.h";
+
+# The maximum number of codepoints that can result from case mapping
+# of a single character. See Unicode section 5.18 "Case Mappings".
+my $MAX_CASE_EXPANSION = 3;
+
+my $FH;
+
+my %simple = ();
+
+open($FH, '<', "$output_path/UnicodeData.txt")
+ or die "Could not open $output_path/UnicodeData.txt: $!.";
+while (my $line = <$FH>)
+{
+ my @elts = split(';', $line);
+ my $code = hex($elts[0]);
+ my $simple_uppercase = hex($elts[12] =~ s/^\s+|\s+$//rg);
+ my $simple_lowercase = hex($elts[13] =~ s/^\s+|\s+$//rg);
+ my $simple_titlecase = hex($elts[14] =~ s/^\s+|\s+$//rg);
+
+ die "codepoint $code out of range" if $code > 0x10FFFF;
+ die "Simple_Lowercase $code out of range" if $simple_lowercase > 0x10FFFF;
+ die "Simple_Titlecase $code out of range" if $simple_titlecase > 0x10FFFF;
+ die "Simple_Uppercase $code out of range" if $simple_uppercase > 0x10FFFF;
+
+ if ($simple_lowercase || $simple_titlecase || $simple_uppercase)
+ {
+ $simple{$code} = {Simple_Lowercase => ($simple_lowercase || $code),
+ Simple_Titlecase => ($simple_titlecase || $code),
+ Simple_Uppercase => ($simple_uppercase || $code)};
+ }
+}
+close $FH;
+
+# Start writing out the output files
+open my $OT, '>', $output_table_file
+ or die "Could not open output file $output_table_file: $!\n";
+
+# determine size of array given that codepoints <= 0x80 are dense and
+# the rest of the entries are sparse
+my $num_simple = 0x80;
+foreach my $code (sort { $a <=> $b } (keys %simple))
+{
+ $num_simple++ unless $code < 0x80;
+}
+
+print $OT <<"HEADER";
+/*-------------------------------------------------------------------------
+ *
+ * unicode_case_table.h
+ * Case mapping and information table.
+ *
+ * Portions Copyright (c) 1996-2023, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/common/unicode_case_table.h
+ *
+ *-------------------------------------------------------------------------
+ */
+
+/*
+ * File auto-generated by src/common/unicode/generate-unicode_case_table.pl,
+ * do not edit. There is deliberately not an #ifndef PG_UNICODE_CASE_TABLE_H
+ * here.
+ */
+
+#include "mb/pg_wchar.h"
+
+typedef struct
+{
+ pg_wchar codepoint; /* Unicode codepoint */
+ pg_wchar simple_lowercase;
+ pg_wchar simple_titlecase;
+ pg_wchar simple_uppercase;
+} pg_simple_case_map;
+
+/*
+ * Case mapping table. Dense for codepoints < 0x80 (enabling fast lookup),
+ * sparse for higher codepoints (requiring scan or binary search).
+ */
+static const pg_simple_case_map simple_case_map[$num_simple] =
+{
+HEADER
+
+my $firsttime = 1;
+
+printf $OT "\t/* begin dense entries for codepoints < 0x80 */\n";
+for (my $code = 0; $code < 0x80; $code++)
+{
+ printf $OT ",\n" unless $firsttime;
+ $firsttime = 0;
+
+ my $lc = ($simple{$code}{Simple_Lowercase} || $code);
+ my $tc = ($simple{$code}{Simple_Titlecase} || $code);
+ my $uc = ($simple{$code}{Simple_Uppercase} || $code);
+ printf $OT "\t{0x%06x, 0x%06x, 0x%06x, 0x%06x}", $code, $lc, $tc, $uc;
+}
+printf $OT ",\n\n";
+
+$firsttime = 1;
+printf $OT "\t/* begin sparse entries for codepoints >= 0x80 */\n";
+foreach my $code (sort { $a <=> $b } (keys %simple))
+{
+ next unless $code >= 0x80; # already output above
+
+ printf $OT ",\n" unless $firsttime;
+ $firsttime = 0;
+
+ my $map = $simple{$code};
+ printf $OT "\t{0x%06x, 0x%06x, 0x%06x, 0x%06x}",
+ $code, $map->{Simple_Lowercase}, $map->{Simple_Titlecase}, $map->{Simple_Uppercase};
+}
+print $OT "\n};\n";
diff --git a/src/common/unicode/meson.build b/src/common/unicode/meson.build
index 3526ddb846..85ff40fad9 100644
--- a/src/common/unicode/meson.build
+++ b/src/common/unicode/meson.build
@@ -24,6 +24,16 @@ endforeach
update_unicode_targets = []
+update_unicode_targets += \
+ custom_target('unicode_case_table.h',
+ input: [unicode_data['UnicodeData.txt']],
+ output: ['unicode_case_table.h'],
+ command: [
+ perl, files('generate-unicode_case_table.pl'),
+ '--outdir', '@OUTDIR@', '@INPUT@'],
+ build_by_default: false,
+ )
+
update_unicode_targets += \
custom_target('unicode_category_table.h',
input: [unicode_data['UnicodeData.txt'], unicode_data['DerivedCoreProperties.txt'], unicode_data['PropList.txt']],
@@ -92,6 +102,17 @@ norm_test_table = custom_target('norm_test_table.h',
inc = include_directories('.')
+case_test = executable('case_test',
+ ['case_test.c'],
+ dependencies: [frontend_port_code, icu],
+ include_directories: inc,
+ link_with: [common_static, pgport_static],
+ build_by_default: false,
+ kwargs: default_bin_args + {
+ 'install': false,
+ }
+)
+
category_test = executable('category_test',
['category_test.c'],
dependencies: [frontend_port_code, icu],
@@ -116,6 +137,16 @@ norm_test = executable('norm_test',
update_unicode_dep = []
+if not meson.is_cross_build()
+ update_unicode_dep += custom_target('case_test.run',
+ output: 'case_test.run',
+ input: update_unicode_targets,
+ command: [case_test, UNICODE_VERSION],
+ build_by_default: false,
+ build_always_stale: true,
+ )
+endif
+
if not meson.is_cross_build()
update_unicode_dep += custom_target('category_test.run',
output: 'category_test.run',
diff --git a/src/common/unicode_case.c b/src/common/unicode_case.c
new file mode 100644
index 0000000000..714e0453e1
--- /dev/null
+++ b/src/common/unicode_case.c
@@ -0,0 +1,87 @@
+/*-------------------------------------------------------------------------
+ * unicode_case.c
+ * Conversion to upper or lower case.
+ *
+ * Portions Copyright (c) 2017-2023, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/common/unicode_case.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef FRONTEND
+#include "postgres.h"
+#else
+#include "postgres_fe.h"
+#endif
+
+#include "common/unicode_case.h"
+#include "common/unicode_case_table.h"
+
+/* find entry in simple case map, if any */
+static inline const pg_simple_case_map *
+find_case_map(pg_wchar ucs)
+{
+ int min = 0;
+ int mid;
+ int max = lengthof(simple_case_map) - 1;
+
+ /* all chars <= 0x80 are stored in array for fast lookup */
+ Assert(max >= 0x7f);
+ if (ucs < 0x80)
+ {
+ const pg_simple_case_map *map = &simple_case_map[ucs];
+ Assert(map->codepoint == ucs);
+ return map;
+ }
+
+ /* otherwise, binary search */
+ while (max >= min)
+ {
+ mid = (min + max) / 2;
+ if (ucs > simple_case_map[mid].codepoint)
+ min = mid + 1;
+ else if (ucs < simple_case_map[mid].codepoint)
+ max = mid - 1;
+ else
+ return &simple_case_map[mid];
+ }
+
+ return NULL;
+}
+
+/*
+ * Returns simple lowercase mapping for the given character, or the original
+ * character if none. Sets *special to the special case mapping, if any.
+ */
+pg_wchar
+unicode_lowercase_simple(pg_wchar ucs)
+{
+ const pg_simple_case_map *map = find_case_map(ucs);
+
+ return map ? map->simple_lowercase : ucs;
+}
+
+/*
+ * Returns simple titlecase mapping for the given character, or the original
+ * character if none. Sets *special to the special case mapping, if any.
+ */
+pg_wchar
+unicode_titlecase_simple(pg_wchar ucs)
+{
+ const pg_simple_case_map *map = find_case_map(ucs);
+
+ return map ? map->simple_titlecase : ucs;
+}
+
+/*
+ * Returns simple uppercase mapping for the given character, or the original
+ * character if none. Sets *special to the special case mapping, if any.
+ */
+pg_wchar
+unicode_uppercase_simple(pg_wchar ucs)
+{
+ const pg_simple_case_map *map = find_case_map(ucs);
+
+ return map ? map->simple_uppercase : ucs;
+}
diff --git a/src/include/common/unicode_case.h b/src/include/common/unicode_case.h
new file mode 100644
index 0000000000..fe5fe6af63
--- /dev/null
+++ b/src/include/common/unicode_case.h
@@ -0,0 +1,23 @@
+/*-------------------------------------------------------------------------
+ *
+ * unicode_case.h
+ * Routines for converting character case.
+ *
+ * These definitions can be used by both frontend and backend code.
+ *
+ * Copyright (c) 2017-2023, PostgreSQL Global Development Group
+ *
+ * src/include/common/unicode_case.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef UNICODE_CASE_H
+#define UNICODE_CASE_H
+
+#include "mb/pg_wchar.h"
+
+pg_wchar unicode_lowercase_simple(pg_wchar ucs);
+pg_wchar unicode_titlecase_simple(pg_wchar ucs);
+pg_wchar unicode_uppercase_simple(pg_wchar ucs);
+
+#endif /* UNICODE_CASE_H */
diff --git a/src/include/common/unicode_case_table.h b/src/include/common/unicode_case_table.h
new file mode 100644
index 0000000000..7733457bec
--- /dev/null
+++ b/src/include/common/unicode_case_table.h
@@ -0,0 +1,2994 @@
+/*-------------------------------------------------------------------------
+ *
+ * unicode_case_table.h
+ * Case mapping and information table.
+ *
+ * Portions Copyright (c) 1996-2023, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/common/unicode_case_table.h
+ *
+ *-------------------------------------------------------------------------
+ */
+
+/*
+ * File auto-generated by src/common/unicode/generate-unicode_case_table.pl,
+ * do not edit. There is deliberately not an #ifndef PG_UNICODE_CASE_TABLE_H
+ * here.
+ */
+
+#include "mb/pg_wchar.h"
+
+typedef struct
+{
+ pg_wchar codepoint; /* Unicode codepoint */
+ pg_wchar simple_lowercase;
+ pg_wchar simple_titlecase;
+ pg_wchar simple_uppercase;
+} pg_simple_case_map;
+
+/*
+ * Case mapping table. Dense for codepoints < 0x80 (enabling fast lookup),
+ * sparse for higher codepoints (requiring scan or binary search).
+ */
+static const pg_simple_case_map simple_case_map[2955] =
+{
+ /* begin dense entries for codepoints < 0x80 */
+ {0x000000, 0x000000, 0x000000, 0x000000},
+ {0x000001, 0x000001, 0x000001, 0x000001},
+ {0x000002, 0x000002, 0x000002, 0x000002},
+ {0x000003, 0x000003, 0x000003, 0x000003},
+ {0x000004, 0x000004, 0x000004, 0x000004},
+ {0x000005, 0x000005, 0x000005, 0x000005},
+ {0x000006, 0x000006, 0x000006, 0x000006},
+ {0x000007, 0x000007, 0x000007, 0x000007},
+ {0x000008, 0x000008, 0x000008, 0x000008},
+ {0x000009, 0x000009, 0x000009, 0x000009},
+ {0x00000a, 0x00000a, 0x00000a, 0x00000a},
+ {0x00000b, 0x00000b, 0x00000b, 0x00000b},
+ {0x00000c, 0x00000c, 0x00000c, 0x00000c},
+ {0x00000d, 0x00000d, 0x00000d, 0x00000d},
+ {0x00000e, 0x00000e, 0x00000e, 0x00000e},
+ {0x00000f, 0x00000f, 0x00000f, 0x00000f},
+ {0x000010, 0x000010, 0x000010, 0x000010},
+ {0x000011, 0x000011, 0x000011, 0x000011},
+ {0x000012, 0x000012, 0x000012, 0x000012},
+ {0x000013, 0x000013, 0x000013, 0x000013},
+ {0x000014, 0x000014, 0x000014, 0x000014},
+ {0x000015, 0x000015, 0x000015, 0x000015},
+ {0x000016, 0x000016, 0x000016, 0x000016},
+ {0x000017, 0x000017, 0x000017, 0x000017},
+ {0x000018, 0x000018, 0x000018, 0x000018},
+ {0x000019, 0x000019, 0x000019, 0x000019},
+ {0x00001a, 0x00001a, 0x00001a, 0x00001a},
+ {0x00001b, 0x00001b, 0x00001b, 0x00001b},
+ {0x00001c, 0x00001c, 0x00001c, 0x00001c},
+ {0x00001d, 0x00001d, 0x00001d, 0x00001d},
+ {0x00001e, 0x00001e, 0x00001e, 0x00001e},
+ {0x00001f, 0x00001f, 0x00001f, 0x00001f},
+ {0x000020, 0x000020, 0x000020, 0x000020},
+ {0x000021, 0x000021, 0x000021, 0x000021},
+ {0x000022, 0x000022, 0x000022, 0x000022},
+ {0x000023, 0x000023, 0x000023, 0x000023},
+ {0x000024, 0x000024, 0x000024, 0x000024},
+ {0x000025, 0x000025, 0x000025, 0x000025},
+ {0x000026, 0x000026, 0x000026, 0x000026},
+ {0x000027, 0x000027, 0x000027, 0x000027},
+ {0x000028, 0x000028, 0x000028, 0x000028},
+ {0x000029, 0x000029, 0x000029, 0x000029},
+ {0x00002a, 0x00002a, 0x00002a, 0x00002a},
+ {0x00002b, 0x00002b, 0x00002b, 0x00002b},
+ {0x00002c, 0x00002c, 0x00002c, 0x00002c},
+ {0x00002d, 0x00002d, 0x00002d, 0x00002d},
+ {0x00002e, 0x00002e, 0x00002e, 0x00002e},
+ {0x00002f, 0x00002f, 0x00002f, 0x00002f},
+ {0x000030, 0x000030, 0x000030, 0x000030},
+ {0x000031, 0x000031, 0x000031, 0x000031},
+ {0x000032, 0x000032, 0x000032, 0x000032},
+ {0x000033, 0x000033, 0x000033, 0x000033},
+ {0x000034, 0x000034, 0x000034, 0x000034},
+ {0x000035, 0x000035, 0x000035, 0x000035},
+ {0x000036, 0x000036, 0x000036, 0x000036},
+ {0x000037, 0x000037, 0x000037, 0x000037},
+ {0x000038, 0x000038, 0x000038, 0x000038},
+ {0x000039, 0x000039, 0x000039, 0x000039},
+ {0x00003a, 0x00003a, 0x00003a, 0x00003a},
+ {0x00003b, 0x00003b, 0x00003b, 0x00003b},
+ {0x00003c, 0x00003c, 0x00003c, 0x00003c},
+ {0x00003d, 0x00003d, 0x00003d, 0x00003d},
+ {0x00003e, 0x00003e, 0x00003e, 0x00003e},
+ {0x00003f, 0x00003f, 0x00003f, 0x00003f},
+ {0x000040, 0x000040, 0x000040, 0x000040},
+ {0x000041, 0x000061, 0x000041, 0x000041},
+ {0x000042, 0x000062, 0x000042, 0x000042},
+ {0x000043, 0x000063, 0x000043, 0x000043},
+ {0x000044, 0x000064, 0x000044, 0x000044},
+ {0x000045, 0x000065, 0x000045, 0x000045},
+ {0x000046, 0x000066, 0x000046, 0x000046},
+ {0x000047, 0x000067, 0x000047, 0x000047},
+ {0x000048, 0x000068, 0x000048, 0x000048},
+ {0x000049, 0x000069, 0x000049, 0x000049},
+ {0x00004a, 0x00006a, 0x00004a, 0x00004a},
+ {0x00004b, 0x00006b, 0x00004b, 0x00004b},
+ {0x00004c, 0x00006c, 0x00004c, 0x00004c},
+ {0x00004d, 0x00006d, 0x00004d, 0x00004d},
+ {0x00004e, 0x00006e, 0x00004e, 0x00004e},
+ {0x00004f, 0x00006f, 0x00004f, 0x00004f},
+ {0x000050, 0x000070, 0x000050, 0x000050},
+ {0x000051, 0x000071, 0x000051, 0x000051},
+ {0x000052, 0x000072, 0x000052, 0x000052},
+ {0x000053, 0x000073, 0x000053, 0x000053},
+ {0x000054, 0x000074, 0x000054, 0x000054},
+ {0x000055, 0x000075, 0x000055, 0x000055},
+ {0x000056, 0x000076, 0x000056, 0x000056},
+ {0x000057, 0x000077, 0x000057, 0x000057},
+ {0x000058, 0x000078, 0x000058, 0x000058},
+ {0x000059, 0x000079, 0x000059, 0x000059},
+ {0x00005a, 0x00007a, 0x00005a, 0x00005a},
+ {0x00005b, 0x00005b, 0x00005b, 0x00005b},
+ {0x00005c, 0x00005c, 0x00005c, 0x00005c},
+ {0x00005d, 0x00005d, 0x00005d, 0x00005d},
+ {0x00005e, 0x00005e, 0x00005e, 0x00005e},
+ {0x00005f, 0x00005f, 0x00005f, 0x00005f},
+ {0x000060, 0x000060, 0x000060, 0x000060},
+ {0x000061, 0x000061, 0x000041, 0x000041},
+ {0x000062, 0x000062, 0x000042, 0x000042},
+ {0x000063, 0x000063, 0x000043, 0x000043},
+ {0x000064, 0x000064, 0x000044, 0x000044},
+ {0x000065, 0x000065, 0x000045, 0x000045},
+ {0x000066, 0x000066, 0x000046, 0x000046},
+ {0x000067, 0x000067, 0x000047, 0x000047},
+ {0x000068, 0x000068, 0x000048, 0x000048},
+ {0x000069, 0x000069, 0x000049, 0x000049},
+ {0x00006a, 0x00006a, 0x00004a, 0x00004a},
+ {0x00006b, 0x00006b, 0x00004b, 0x00004b},
+ {0x00006c, 0x00006c, 0x00004c, 0x00004c},
+ {0x00006d, 0x00006d, 0x00004d, 0x00004d},
+ {0x00006e, 0x00006e, 0x00004e, 0x00004e},
+ {0x00006f, 0x00006f, 0x00004f, 0x00004f},
+ {0x000070, 0x000070, 0x000050, 0x000050},
+ {0x000071, 0x000071, 0x000051, 0x000051},
+ {0x000072, 0x000072, 0x000052, 0x000052},
+ {0x000073, 0x000073, 0x000053, 0x000053},
+ {0x000074, 0x000074, 0x000054, 0x000054},
+ {0x000075, 0x000075, 0x000055, 0x000055},
+ {0x000076, 0x000076, 0x000056, 0x000056},
+ {0x000077, 0x000077, 0x000057, 0x000057},
+ {0x000078, 0x000078, 0x000058, 0x000058},
+ {0x000079, 0x000079, 0x000059, 0x000059},
+ {0x00007a, 0x00007a, 0x00005a, 0x00005a},
+ {0x00007b, 0x00007b, 0x00007b, 0x00007b},
+ {0x00007c, 0x00007c, 0x00007c, 0x00007c},
+ {0x00007d, 0x00007d, 0x00007d, 0x00007d},
+ {0x00007e, 0x00007e, 0x00007e, 0x00007e},
+ {0x00007f, 0x00007f, 0x00007f, 0x00007f},
+
+ /* begin sparse entries for codepoints >= 0x80 */
+ {0x0000b5, 0x0000b5, 0x00039c, 0x00039c},
+ {0x0000c0, 0x0000e0, 0x0000c0, 0x0000c0},
+ {0x0000c1, 0x0000e1, 0x0000c1, 0x0000c1},
+ {0x0000c2, 0x0000e2, 0x0000c2, 0x0000c2},
+ {0x0000c3, 0x0000e3, 0x0000c3, 0x0000c3},
+ {0x0000c4, 0x0000e4, 0x0000c4, 0x0000c4},
+ {0x0000c5, 0x0000e5, 0x0000c5, 0x0000c5},
+ {0x0000c6, 0x0000e6, 0x0000c6, 0x0000c6},
+ {0x0000c7, 0x0000e7, 0x0000c7, 0x0000c7},
+ {0x0000c8, 0x0000e8, 0x0000c8, 0x0000c8},
+ {0x0000c9, 0x0000e9, 0x0000c9, 0x0000c9},
+ {0x0000ca, 0x0000ea, 0x0000ca, 0x0000ca},
+ {0x0000cb, 0x0000eb, 0x0000cb, 0x0000cb},
+ {0x0000cc, 0x0000ec, 0x0000cc, 0x0000cc},
+ {0x0000cd, 0x0000ed, 0x0000cd, 0x0000cd},
+ {0x0000ce, 0x0000ee, 0x0000ce, 0x0000ce},
+ {0x0000cf, 0x0000ef, 0x0000cf, 0x0000cf},
+ {0x0000d0, 0x0000f0, 0x0000d0, 0x0000d0},
+ {0x0000d1, 0x0000f1, 0x0000d1, 0x0000d1},
+ {0x0000d2, 0x0000f2, 0x0000d2, 0x0000d2},
+ {0x0000d3, 0x0000f3, 0x0000d3, 0x0000d3},
+ {0x0000d4, 0x0000f4, 0x0000d4, 0x0000d4},
+ {0x0000d5, 0x0000f5, 0x0000d5, 0x0000d5},
+ {0x0000d6, 0x0000f6, 0x0000d6, 0x0000d6},
+ {0x0000d8, 0x0000f8, 0x0000d8, 0x0000d8},
+ {0x0000d9, 0x0000f9, 0x0000d9, 0x0000d9},
+ {0x0000da, 0x0000fa, 0x0000da, 0x0000da},
+ {0x0000db, 0x0000fb, 0x0000db, 0x0000db},
+ {0x0000dc, 0x0000fc, 0x0000dc, 0x0000dc},
+ {0x0000dd, 0x0000fd, 0x0000dd, 0x0000dd},
+ {0x0000de, 0x0000fe, 0x0000de, 0x0000de},
+ {0x0000e0, 0x0000e0, 0x0000c0, 0x0000c0},
+ {0x0000e1, 0x0000e1, 0x0000c1, 0x0000c1},
+ {0x0000e2, 0x0000e2, 0x0000c2, 0x0000c2},
+ {0x0000e3, 0x0000e3, 0x0000c3, 0x0000c3},
+ {0x0000e4, 0x0000e4, 0x0000c4, 0x0000c4},
+ {0x0000e5, 0x0000e5, 0x0000c5, 0x0000c5},
+ {0x0000e6, 0x0000e6, 0x0000c6, 0x0000c6},
+ {0x0000e7, 0x0000e7, 0x0000c7, 0x0000c7},
+ {0x0000e8, 0x0000e8, 0x0000c8, 0x0000c8},
+ {0x0000e9, 0x0000e9, 0x0000c9, 0x0000c9},
+ {0x0000ea, 0x0000ea, 0x0000ca, 0x0000ca},
+ {0x0000eb, 0x0000eb, 0x0000cb, 0x0000cb},
+ {0x0000ec, 0x0000ec, 0x0000cc, 0x0000cc},
+ {0x0000ed, 0x0000ed, 0x0000cd, 0x0000cd},
+ {0x0000ee, 0x0000ee, 0x0000ce, 0x0000ce},
+ {0x0000ef, 0x0000ef, 0x0000cf, 0x0000cf},
+ {0x0000f0, 0x0000f0, 0x0000d0, 0x0000d0},
+ {0x0000f1, 0x0000f1, 0x0000d1, 0x0000d1},
+ {0x0000f2, 0x0000f2, 0x0000d2, 0x0000d2},
+ {0x0000f3, 0x0000f3, 0x0000d3, 0x0000d3},
+ {0x0000f4, 0x0000f4, 0x0000d4, 0x0000d4},
+ {0x0000f5, 0x0000f5, 0x0000d5, 0x0000d5},
+ {0x0000f6, 0x0000f6, 0x0000d6, 0x0000d6},
+ {0x0000f8, 0x0000f8, 0x0000d8, 0x0000d8},
+ {0x0000f9, 0x0000f9, 0x0000d9, 0x0000d9},
+ {0x0000fa, 0x0000fa, 0x0000da, 0x0000da},
+ {0x0000fb, 0x0000fb, 0x0000db, 0x0000db},
+ {0x0000fc, 0x0000fc, 0x0000dc, 0x0000dc},
+ {0x0000fd, 0x0000fd, 0x0000dd, 0x0000dd},
+ {0x0000fe, 0x0000fe, 0x0000de, 0x0000de},
+ {0x0000ff, 0x0000ff, 0x000178, 0x000178},
+ {0x000100, 0x000101, 0x000100, 0x000100},
+ {0x000101, 0x000101, 0x000100, 0x000100},
+ {0x000102, 0x000103, 0x000102, 0x000102},
+ {0x000103, 0x000103, 0x000102, 0x000102},
+ {0x000104, 0x000105, 0x000104, 0x000104},
+ {0x000105, 0x000105, 0x000104, 0x000104},
+ {0x000106, 0x000107, 0x000106, 0x000106},
+ {0x000107, 0x000107, 0x000106, 0x000106},
+ {0x000108, 0x000109, 0x000108, 0x000108},
+ {0x000109, 0x000109, 0x000108, 0x000108},
+ {0x00010a, 0x00010b, 0x00010a, 0x00010a},
+ {0x00010b, 0x00010b, 0x00010a, 0x00010a},
+ {0x00010c, 0x00010d, 0x00010c, 0x00010c},
+ {0x00010d, 0x00010d, 0x00010c, 0x00010c},
+ {0x00010e, 0x00010f, 0x00010e, 0x00010e},
+ {0x00010f, 0x00010f, 0x00010e, 0x00010e},
+ {0x000110, 0x000111, 0x000110, 0x000110},
+ {0x000111, 0x000111, 0x000110, 0x000110},
+ {0x000112, 0x000113, 0x000112, 0x000112},
+ {0x000113, 0x000113, 0x000112, 0x000112},
+ {0x000114, 0x000115, 0x000114, 0x000114},
+ {0x000115, 0x000115, 0x000114, 0x000114},
+ {0x000116, 0x000117, 0x000116, 0x000116},
+ {0x000117, 0x000117, 0x000116, 0x000116},
+ {0x000118, 0x000119, 0x000118, 0x000118},
+ {0x000119, 0x000119, 0x000118, 0x000118},
+ {0x00011a, 0x00011b, 0x00011a, 0x00011a},
+ {0x00011b, 0x00011b, 0x00011a, 0x00011a},
+ {0x00011c, 0x00011d, 0x00011c, 0x00011c},
+ {0x00011d, 0x00011d, 0x00011c, 0x00011c},
+ {0x00011e, 0x00011f, 0x00011e, 0x00011e},
+ {0x00011f, 0x00011f, 0x00011e, 0x00011e},
+ {0x000120, 0x000121, 0x000120, 0x000120},
+ {0x000121, 0x000121, 0x000120, 0x000120},
+ {0x000122, 0x000123, 0x000122, 0x000122},
+ {0x000123, 0x000123, 0x000122, 0x000122},
+ {0x000124, 0x000125, 0x000124, 0x000124},
+ {0x000125, 0x000125, 0x000124, 0x000124},
+ {0x000126, 0x000127, 0x000126, 0x000126},
+ {0x000127, 0x000127, 0x000126, 0x000126},
+ {0x000128, 0x000129, 0x000128, 0x000128},
+ {0x000129, 0x000129, 0x000128, 0x000128},
+ {0x00012a, 0x00012b, 0x00012a, 0x00012a},
+ {0x00012b, 0x00012b, 0x00012a, 0x00012a},
+ {0x00012c, 0x00012d, 0x00012c, 0x00012c},
+ {0x00012d, 0x00012d, 0x00012c, 0x00012c},
+ {0x00012e, 0x00012f, 0x00012e, 0x00012e},
+ {0x00012f, 0x00012f, 0x00012e, 0x00012e},
+ {0x000130, 0x000069, 0x000130, 0x000130},
+ {0x000131, 0x000131, 0x000049, 0x000049},
+ {0x000132, 0x000133, 0x000132, 0x000132},
+ {0x000133, 0x000133, 0x000132, 0x000132},
+ {0x000134, 0x000135, 0x000134, 0x000134},
+ {0x000135, 0x000135, 0x000134, 0x000134},
+ {0x000136, 0x000137, 0x000136, 0x000136},
+ {0x000137, 0x000137, 0x000136, 0x000136},
+ {0x000139, 0x00013a, 0x000139, 0x000139},
+ {0x00013a, 0x00013a, 0x000139, 0x000139},
+ {0x00013b, 0x00013c, 0x00013b, 0x00013b},
+ {0x00013c, 0x00013c, 0x00013b, 0x00013b},
+ {0x00013d, 0x00013e, 0x00013d, 0x00013d},
+ {0x00013e, 0x00013e, 0x00013d, 0x00013d},
+ {0x00013f, 0x000140, 0x00013f, 0x00013f},
+ {0x000140, 0x000140, 0x00013f, 0x00013f},
+ {0x000141, 0x000142, 0x000141, 0x000141},
+ {0x000142, 0x000142, 0x000141, 0x000141},
+ {0x000143, 0x000144, 0x000143, 0x000143},
+ {0x000144, 0x000144, 0x000143, 0x000143},
+ {0x000145, 0x000146, 0x000145, 0x000145},
+ {0x000146, 0x000146, 0x000145, 0x000145},
+ {0x000147, 0x000148, 0x000147, 0x000147},
+ {0x000148, 0x000148, 0x000147, 0x000147},
+ {0x00014a, 0x00014b, 0x00014a, 0x00014a},
+ {0x00014b, 0x00014b, 0x00014a, 0x00014a},
+ {0x00014c, 0x00014d, 0x00014c, 0x00014c},
+ {0x00014d, 0x00014d, 0x00014c, 0x00014c},
+ {0x00014e, 0x00014f, 0x00014e, 0x00014e},
+ {0x00014f, 0x00014f, 0x00014e, 0x00014e},
+ {0x000150, 0x000151, 0x000150, 0x000150},
+ {0x000151, 0x000151, 0x000150, 0x000150},
+ {0x000152, 0x000153, 0x000152, 0x000152},
+ {0x000153, 0x000153, 0x000152, 0x000152},
+ {0x000154, 0x000155, 0x000154, 0x000154},
+ {0x000155, 0x000155, 0x000154, 0x000154},
+ {0x000156, 0x000157, 0x000156, 0x000156},
+ {0x000157, 0x000157, 0x000156, 0x000156},
+ {0x000158, 0x000159, 0x000158, 0x000158},
+ {0x000159, 0x000159, 0x000158, 0x000158},
+ {0x00015a, 0x00015b, 0x00015a, 0x00015a},
+ {0x00015b, 0x00015b, 0x00015a, 0x00015a},
+ {0x00015c, 0x00015d, 0x00015c, 0x00015c},
+ {0x00015d, 0x00015d, 0x00015c, 0x00015c},
+ {0x00015e, 0x00015f, 0x00015e, 0x00015e},
+ {0x00015f, 0x00015f, 0x00015e, 0x00015e},
+ {0x000160, 0x000161, 0x000160, 0x000160},
+ {0x000161, 0x000161, 0x000160, 0x000160},
+ {0x000162, 0x000163, 0x000162, 0x000162},
+ {0x000163, 0x000163, 0x000162, 0x000162},
+ {0x000164, 0x000165, 0x000164, 0x000164},
+ {0x000165, 0x000165, 0x000164, 0x000164},
+ {0x000166, 0x000167, 0x000166, 0x000166},
+ {0x000167, 0x000167, 0x000166, 0x000166},
+ {0x000168, 0x000169, 0x000168, 0x000168},
+ {0x000169, 0x000169, 0x000168, 0x000168},
+ {0x00016a, 0x00016b, 0x00016a, 0x00016a},
+ {0x00016b, 0x00016b, 0x00016a, 0x00016a},
+ {0x00016c, 0x00016d, 0x00016c, 0x00016c},
+ {0x00016d, 0x00016d, 0x00016c, 0x00016c},
+ {0x00016e, 0x00016f, 0x00016e, 0x00016e},
+ {0x00016f, 0x00016f, 0x00016e, 0x00016e},
+ {0x000170, 0x000171, 0x000170, 0x000170},
+ {0x000171, 0x000171, 0x000170, 0x000170},
+ {0x000172, 0x000173, 0x000172, 0x000172},
+ {0x000173, 0x000173, 0x000172, 0x000172},
+ {0x000174, 0x000175, 0x000174, 0x000174},
+ {0x000175, 0x000175, 0x000174, 0x000174},
+ {0x000176, 0x000177, 0x000176, 0x000176},
+ {0x000177, 0x000177, 0x000176, 0x000176},
+ {0x000178, 0x0000ff, 0x000178, 0x000178},
+ {0x000179, 0x00017a, 0x000179, 0x000179},
+ {0x00017a, 0x00017a, 0x000179, 0x000179},
+ {0x00017b, 0x00017c, 0x00017b, 0x00017b},
+ {0x00017c, 0x00017c, 0x00017b, 0x00017b},
+ {0x00017d, 0x00017e, 0x00017d, 0x00017d},
+ {0x00017e, 0x00017e, 0x00017d, 0x00017d},
+ {0x00017f, 0x00017f, 0x000053, 0x000053},
+ {0x000180, 0x000180, 0x000243, 0x000243},
+ {0x000181, 0x000253, 0x000181, 0x000181},
+ {0x000182, 0x000183, 0x000182, 0x000182},
+ {0x000183, 0x000183, 0x000182, 0x000182},
+ {0x000184, 0x000185, 0x000184, 0x000184},
+ {0x000185, 0x000185, 0x000184, 0x000184},
+ {0x000186, 0x000254, 0x000186, 0x000186},
+ {0x000187, 0x000188, 0x000187, 0x000187},
+ {0x000188, 0x000188, 0x000187, 0x000187},
+ {0x000189, 0x000256, 0x000189, 0x000189},
+ {0x00018a, 0x000257, 0x00018a, 0x00018a},
+ {0x00018b, 0x00018c, 0x00018b, 0x00018b},
+ {0x00018c, 0x00018c, 0x00018b, 0x00018b},
+ {0x00018e, 0x0001dd, 0x00018e, 0x00018e},
+ {0x00018f, 0x000259, 0x00018f, 0x00018f},
+ {0x000190, 0x00025b, 0x000190, 0x000190},
+ {0x000191, 0x000192, 0x000191, 0x000191},
+ {0x000192, 0x000192, 0x000191, 0x000191},
+ {0x000193, 0x000260, 0x000193, 0x000193},
+ {0x000194, 0x000263, 0x000194, 0x000194},
+ {0x000195, 0x000195, 0x0001f6, 0x0001f6},
+ {0x000196, 0x000269, 0x000196, 0x000196},
+ {0x000197, 0x000268, 0x000197, 0x000197},
+ {0x000198, 0x000199, 0x000198, 0x000198},
+ {0x000199, 0x000199, 0x000198, 0x000198},
+ {0x00019a, 0x00019a, 0x00023d, 0x00023d},
+ {0x00019c, 0x00026f, 0x00019c, 0x00019c},
+ {0x00019d, 0x000272, 0x00019d, 0x00019d},
+ {0x00019e, 0x00019e, 0x000220, 0x000220},
+ {0x00019f, 0x000275, 0x00019f, 0x00019f},
+ {0x0001a0, 0x0001a1, 0x0001a0, 0x0001a0},
+ {0x0001a1, 0x0001a1, 0x0001a0, 0x0001a0},
+ {0x0001a2, 0x0001a3, 0x0001a2, 0x0001a2},
+ {0x0001a3, 0x0001a3, 0x0001a2, 0x0001a2},
+ {0x0001a4, 0x0001a5, 0x0001a4, 0x0001a4},
+ {0x0001a5, 0x0001a5, 0x0001a4, 0x0001a4},
+ {0x0001a6, 0x000280, 0x0001a6, 0x0001a6},
+ {0x0001a7, 0x0001a8, 0x0001a7, 0x0001a7},
+ {0x0001a8, 0x0001a8, 0x0001a7, 0x0001a7},
+ {0x0001a9, 0x000283, 0x0001a9, 0x0001a9},
+ {0x0001ac, 0x0001ad, 0x0001ac, 0x0001ac},
+ {0x0001ad, 0x0001ad, 0x0001ac, 0x0001ac},
+ {0x0001ae, 0x000288, 0x0001ae, 0x0001ae},
+ {0x0001af, 0x0001b0, 0x0001af, 0x0001af},
+ {0x0001b0, 0x0001b0, 0x0001af, 0x0001af},
+ {0x0001b1, 0x00028a, 0x0001b1, 0x0001b1},
+ {0x0001b2, 0x00028b, 0x0001b2, 0x0001b2},
+ {0x0001b3, 0x0001b4, 0x0001b3, 0x0001b3},
+ {0x0001b4, 0x0001b4, 0x0001b3, 0x0001b3},
+ {0x0001b5, 0x0001b6, 0x0001b5, 0x0001b5},
+ {0x0001b6, 0x0001b6, 0x0001b5, 0x0001b5},
+ {0x0001b7, 0x000292, 0x0001b7, 0x0001b7},
+ {0x0001b8, 0x0001b9, 0x0001b8, 0x0001b8},
+ {0x0001b9, 0x0001b9, 0x0001b8, 0x0001b8},
+ {0x0001bc, 0x0001bd, 0x0001bc, 0x0001bc},
+ {0x0001bd, 0x0001bd, 0x0001bc, 0x0001bc},
+ {0x0001bf, 0x0001bf, 0x0001f7, 0x0001f7},
+ {0x0001c4, 0x0001c6, 0x0001c5, 0x0001c4},
+ {0x0001c5, 0x0001c6, 0x0001c5, 0x0001c4},
+ {0x0001c6, 0x0001c6, 0x0001c5, 0x0001c4},
+ {0x0001c7, 0x0001c9, 0x0001c8, 0x0001c7},
+ {0x0001c8, 0x0001c9, 0x0001c8, 0x0001c7},
+ {0x0001c9, 0x0001c9, 0x0001c8, 0x0001c7},
+ {0x0001ca, 0x0001cc, 0x0001cb, 0x0001ca},
+ {0x0001cb, 0x0001cc, 0x0001cb, 0x0001ca},
+ {0x0001cc, 0x0001cc, 0x0001cb, 0x0001ca},
+ {0x0001cd, 0x0001ce, 0x0001cd, 0x0001cd},
+ {0x0001ce, 0x0001ce, 0x0001cd, 0x0001cd},
+ {0x0001cf, 0x0001d0, 0x0001cf, 0x0001cf},
+ {0x0001d0, 0x0001d0, 0x0001cf, 0x0001cf},
+ {0x0001d1, 0x0001d2, 0x0001d1, 0x0001d1},
+ {0x0001d2, 0x0001d2, 0x0001d1, 0x0001d1},
+ {0x0001d3, 0x0001d4, 0x0001d3, 0x0001d3},
+ {0x0001d4, 0x0001d4, 0x0001d3, 0x0001d3},
+ {0x0001d5, 0x0001d6, 0x0001d5, 0x0001d5},
+ {0x0001d6, 0x0001d6, 0x0001d5, 0x0001d5},
+ {0x0001d7, 0x0001d8, 0x0001d7, 0x0001d7},
+ {0x0001d8, 0x0001d8, 0x0001d7, 0x0001d7},
+ {0x0001d9, 0x0001da, 0x0001d9, 0x0001d9},
+ {0x0001da, 0x0001da, 0x0001d9, 0x0001d9},
+ {0x0001db, 0x0001dc, 0x0001db, 0x0001db},
+ {0x0001dc, 0x0001dc, 0x0001db, 0x0001db},
+ {0x0001dd, 0x0001dd, 0x00018e, 0x00018e},
+ {0x0001de, 0x0001df, 0x0001de, 0x0001de},
+ {0x0001df, 0x0001df, 0x0001de, 0x0001de},
+ {0x0001e0, 0x0001e1, 0x0001e0, 0x0001e0},
+ {0x0001e1, 0x0001e1, 0x0001e0, 0x0001e0},
+ {0x0001e2, 0x0001e3, 0x0001e2, 0x0001e2},
+ {0x0001e3, 0x0001e3, 0x0001e2, 0x0001e2},
+ {0x0001e4, 0x0001e5, 0x0001e4, 0x0001e4},
+ {0x0001e5, 0x0001e5, 0x0001e4, 0x0001e4},
+ {0x0001e6, 0x0001e7, 0x0001e6, 0x0001e6},
+ {0x0001e7, 0x0001e7, 0x0001e6, 0x0001e6},
+ {0x0001e8, 0x0001e9, 0x0001e8, 0x0001e8},
+ {0x0001e9, 0x0001e9, 0x0001e8, 0x0001e8},
+ {0x0001ea, 0x0001eb, 0x0001ea, 0x0001ea},
+ {0x0001eb, 0x0001eb, 0x0001ea, 0x0001ea},
+ {0x0001ec, 0x0001ed, 0x0001ec, 0x0001ec},
+ {0x0001ed, 0x0001ed, 0x0001ec, 0x0001ec},
+ {0x0001ee, 0x0001ef, 0x0001ee, 0x0001ee},
+ {0x0001ef, 0x0001ef, 0x0001ee, 0x0001ee},
+ {0x0001f1, 0x0001f3, 0x0001f2, 0x0001f1},
+ {0x0001f2, 0x0001f3, 0x0001f2, 0x0001f1},
+ {0x0001f3, 0x0001f3, 0x0001f2, 0x0001f1},
+ {0x0001f4, 0x0001f5, 0x0001f4, 0x0001f4},
+ {0x0001f5, 0x0001f5, 0x0001f4, 0x0001f4},
+ {0x0001f6, 0x000195, 0x0001f6, 0x0001f6},
+ {0x0001f7, 0x0001bf, 0x0001f7, 0x0001f7},
+ {0x0001f8, 0x0001f9, 0x0001f8, 0x0001f8},
+ {0x0001f9, 0x0001f9, 0x0001f8, 0x0001f8},
+ {0x0001fa, 0x0001fb, 0x0001fa, 0x0001fa},
+ {0x0001fb, 0x0001fb, 0x0001fa, 0x0001fa},
+ {0x0001fc, 0x0001fd, 0x0001fc, 0x0001fc},
+ {0x0001fd, 0x0001fd, 0x0001fc, 0x0001fc},
+ {0x0001fe, 0x0001ff, 0x0001fe, 0x0001fe},
+ {0x0001ff, 0x0001ff, 0x0001fe, 0x0001fe},
+ {0x000200, 0x000201, 0x000200, 0x000200},
+ {0x000201, 0x000201, 0x000200, 0x000200},
+ {0x000202, 0x000203, 0x000202, 0x000202},
+ {0x000203, 0x000203, 0x000202, 0x000202},
+ {0x000204, 0x000205, 0x000204, 0x000204},
+ {0x000205, 0x000205, 0x000204, 0x000204},
+ {0x000206, 0x000207, 0x000206, 0x000206},
+ {0x000207, 0x000207, 0x000206, 0x000206},
+ {0x000208, 0x000209, 0x000208, 0x000208},
+ {0x000209, 0x000209, 0x000208, 0x000208},
+ {0x00020a, 0x00020b, 0x00020a, 0x00020a},
+ {0x00020b, 0x00020b, 0x00020a, 0x00020a},
+ {0x00020c, 0x00020d, 0x00020c, 0x00020c},
+ {0x00020d, 0x00020d, 0x00020c, 0x00020c},
+ {0x00020e, 0x00020f, 0x00020e, 0x00020e},
+ {0x00020f, 0x00020f, 0x00020e, 0x00020e},
+ {0x000210, 0x000211, 0x000210, 0x000210},
+ {0x000211, 0x000211, 0x000210, 0x000210},
+ {0x000212, 0x000213, 0x000212, 0x000212},
+ {0x000213, 0x000213, 0x000212, 0x000212},
+ {0x000214, 0x000215, 0x000214, 0x000214},
+ {0x000215, 0x000215, 0x000214, 0x000214},
+ {0x000216, 0x000217, 0x000216, 0x000216},
+ {0x000217, 0x000217, 0x000216, 0x000216},
+ {0x000218, 0x000219, 0x000218, 0x000218},
+ {0x000219, 0x000219, 0x000218, 0x000218},
+ {0x00021a, 0x00021b, 0x00021a, 0x00021a},
+ {0x00021b, 0x00021b, 0x00021a, 0x00021a},
+ {0x00021c, 0x00021d, 0x00021c, 0x00021c},
+ {0x00021d, 0x00021d, 0x00021c, 0x00021c},
+ {0x00021e, 0x00021f, 0x00021e, 0x00021e},
+ {0x00021f, 0x00021f, 0x00021e, 0x00021e},
+ {0x000220, 0x00019e, 0x000220, 0x000220},
+ {0x000222, 0x000223, 0x000222, 0x000222},
+ {0x000223, 0x000223, 0x000222, 0x000222},
+ {0x000224, 0x000225, 0x000224, 0x000224},
+ {0x000225, 0x000225, 0x000224, 0x000224},
+ {0x000226, 0x000227, 0x000226, 0x000226},
+ {0x000227, 0x000227, 0x000226, 0x000226},
+ {0x000228, 0x000229, 0x000228, 0x000228},
+ {0x000229, 0x000229, 0x000228, 0x000228},
+ {0x00022a, 0x00022b, 0x00022a, 0x00022a},
+ {0x00022b, 0x00022b, 0x00022a, 0x00022a},
+ {0x00022c, 0x00022d, 0x00022c, 0x00022c},
+ {0x00022d, 0x00022d, 0x00022c, 0x00022c},
+ {0x00022e, 0x00022f, 0x00022e, 0x00022e},
+ {0x00022f, 0x00022f, 0x00022e, 0x00022e},
+ {0x000230, 0x000231, 0x000230, 0x000230},
+ {0x000231, 0x000231, 0x000230, 0x000230},
+ {0x000232, 0x000233, 0x000232, 0x000232},
+ {0x000233, 0x000233, 0x000232, 0x000232},
+ {0x00023a, 0x002c65, 0x00023a, 0x00023a},
+ {0x00023b, 0x00023c, 0x00023b, 0x00023b},
+ {0x00023c, 0x00023c, 0x00023b, 0x00023b},
+ {0x00023d, 0x00019a, 0x00023d, 0x00023d},
+ {0x00023e, 0x002c66, 0x00023e, 0x00023e},
+ {0x00023f, 0x00023f, 0x002c7e, 0x002c7e},
+ {0x000240, 0x000240, 0x002c7f, 0x002c7f},
+ {0x000241, 0x000242, 0x000241, 0x000241},
+ {0x000242, 0x000242, 0x000241, 0x000241},
+ {0x000243, 0x000180, 0x000243, 0x000243},
+ {0x000244, 0x000289, 0x000244, 0x000244},
+ {0x000245, 0x00028c, 0x000245, 0x000245},
+ {0x000246, 0x000247, 0x000246, 0x000246},
+ {0x000247, 0x000247, 0x000246, 0x000246},
+ {0x000248, 0x000249, 0x000248, 0x000248},
+ {0x000249, 0x000249, 0x000248, 0x000248},
+ {0x00024a, 0x00024b, 0x00024a, 0x00024a},
+ {0x00024b, 0x00024b, 0x00024a, 0x00024a},
+ {0x00024c, 0x00024d, 0x00024c, 0x00024c},
+ {0x00024d, 0x00024d, 0x00024c, 0x00024c},
+ {0x00024e, 0x00024f, 0x00024e, 0x00024e},
+ {0x00024f, 0x00024f, 0x00024e, 0x00024e},
+ {0x000250, 0x000250, 0x002c6f, 0x002c6f},
+ {0x000251, 0x000251, 0x002c6d, 0x002c6d},
+ {0x000252, 0x000252, 0x002c70, 0x002c70},
+ {0x000253, 0x000253, 0x000181, 0x000181},
+ {0x000254, 0x000254, 0x000186, 0x000186},
+ {0x000256, 0x000256, 0x000189, 0x000189},
+ {0x000257, 0x000257, 0x00018a, 0x00018a},
+ {0x000259, 0x000259, 0x00018f, 0x00018f},
+ {0x00025b, 0x00025b, 0x000190, 0x000190},
+ {0x00025c, 0x00025c, 0x00a7ab, 0x00a7ab},
+ {0x000260, 0x000260, 0x000193, 0x000193},
+ {0x000261, 0x000261, 0x00a7ac, 0x00a7ac},
+ {0x000263, 0x000263, 0x000194, 0x000194},
+ {0x000265, 0x000265, 0x00a78d, 0x00a78d},
+ {0x000266, 0x000266, 0x00a7aa, 0x00a7aa},
+ {0x000268, 0x000268, 0x000197, 0x000197},
+ {0x000269, 0x000269, 0x000196, 0x000196},
+ {0x00026a, 0x00026a, 0x00a7ae, 0x00a7ae},
+ {0x00026b, 0x00026b, 0x002c62, 0x002c62},
+ {0x00026c, 0x00026c, 0x00a7ad, 0x00a7ad},
+ {0x00026f, 0x00026f, 0x00019c, 0x00019c},
+ {0x000271, 0x000271, 0x002c6e, 0x002c6e},
+ {0x000272, 0x000272, 0x00019d, 0x00019d},
+ {0x000275, 0x000275, 0x00019f, 0x00019f},
+ {0x00027d, 0x00027d, 0x002c64, 0x002c64},
+ {0x000280, 0x000280, 0x0001a6, 0x0001a6},
+ {0x000282, 0x000282, 0x00a7c5, 0x00a7c5},
+ {0x000283, 0x000283, 0x0001a9, 0x0001a9},
+ {0x000287, 0x000287, 0x00a7b1, 0x00a7b1},
+ {0x000288, 0x000288, 0x0001ae, 0x0001ae},
+ {0x000289, 0x000289, 0x000244, 0x000244},
+ {0x00028a, 0x00028a, 0x0001b1, 0x0001b1},
+ {0x00028b, 0x00028b, 0x0001b2, 0x0001b2},
+ {0x00028c, 0x00028c, 0x000245, 0x000245},
+ {0x000292, 0x000292, 0x0001b7, 0x0001b7},
+ {0x00029d, 0x00029d, 0x00a7b2, 0x00a7b2},
+ {0x00029e, 0x00029e, 0x00a7b0, 0x00a7b0},
+ {0x000345, 0x000345, 0x000399, 0x000399},
+ {0x000370, 0x000371, 0x000370, 0x000370},
+ {0x000371, 0x000371, 0x000370, 0x000370},
+ {0x000372, 0x000373, 0x000372, 0x000372},
+ {0x000373, 0x000373, 0x000372, 0x000372},
+ {0x000376, 0x000377, 0x000376, 0x000376},
+ {0x000377, 0x000377, 0x000376, 0x000376},
+ {0x00037b, 0x00037b, 0x0003fd, 0x0003fd},
+ {0x00037c, 0x00037c, 0x0003fe, 0x0003fe},
+ {0x00037d, 0x00037d, 0x0003ff, 0x0003ff},
+ {0x00037f, 0x0003f3, 0x00037f, 0x00037f},
+ {0x000386, 0x0003ac, 0x000386, 0x000386},
+ {0x000388, 0x0003ad, 0x000388, 0x000388},
+ {0x000389, 0x0003ae, 0x000389, 0x000389},
+ {0x00038a, 0x0003af, 0x00038a, 0x00038a},
+ {0x00038c, 0x0003cc, 0x00038c, 0x00038c},
+ {0x00038e, 0x0003cd, 0x00038e, 0x00038e},
+ {0x00038f, 0x0003ce, 0x00038f, 0x00038f},
+ {0x000391, 0x0003b1, 0x000391, 0x000391},
+ {0x000392, 0x0003b2, 0x000392, 0x000392},
+ {0x000393, 0x0003b3, 0x000393, 0x000393},
+ {0x000394, 0x0003b4, 0x000394, 0x000394},
+ {0x000395, 0x0003b5, 0x000395, 0x000395},
+ {0x000396, 0x0003b6, 0x000396, 0x000396},
+ {0x000397, 0x0003b7, 0x000397, 0x000397},
+ {0x000398, 0x0003b8, 0x000398, 0x000398},
+ {0x000399, 0x0003b9, 0x000399, 0x000399},
+ {0x00039a, 0x0003ba, 0x00039a, 0x00039a},
+ {0x00039b, 0x0003bb, 0x00039b, 0x00039b},
+ {0x00039c, 0x0003bc, 0x00039c, 0x00039c},
+ {0x00039d, 0x0003bd, 0x00039d, 0x00039d},
+ {0x00039e, 0x0003be, 0x00039e, 0x00039e},
+ {0x00039f, 0x0003bf, 0x00039f, 0x00039f},
+ {0x0003a0, 0x0003c0, 0x0003a0, 0x0003a0},
+ {0x0003a1, 0x0003c1, 0x0003a1, 0x0003a1},
+ {0x0003a3, 0x0003c3, 0x0003a3, 0x0003a3},
+ {0x0003a4, 0x0003c4, 0x0003a4, 0x0003a4},
+ {0x0003a5, 0x0003c5, 0x0003a5, 0x0003a5},
+ {0x0003a6, 0x0003c6, 0x0003a6, 0x0003a6},
+ {0x0003a7, 0x0003c7, 0x0003a7, 0x0003a7},
+ {0x0003a8, 0x0003c8, 0x0003a8, 0x0003a8},
+ {0x0003a9, 0x0003c9, 0x0003a9, 0x0003a9},
+ {0x0003aa, 0x0003ca, 0x0003aa, 0x0003aa},
+ {0x0003ab, 0x0003cb, 0x0003ab, 0x0003ab},
+ {0x0003ac, 0x0003ac, 0x000386, 0x000386},
+ {0x0003ad, 0x0003ad, 0x000388, 0x000388},
+ {0x0003ae, 0x0003ae, 0x000389, 0x000389},
+ {0x0003af, 0x0003af, 0x00038a, 0x00038a},
+ {0x0003b1, 0x0003b1, 0x000391, 0x000391},
+ {0x0003b2, 0x0003b2, 0x000392, 0x000392},
+ {0x0003b3, 0x0003b3, 0x000393, 0x000393},
+ {0x0003b4, 0x0003b4, 0x000394, 0x000394},
+ {0x0003b5, 0x0003b5, 0x000395, 0x000395},
+ {0x0003b6, 0x0003b6, 0x000396, 0x000396},
+ {0x0003b7, 0x0003b7, 0x000397, 0x000397},
+ {0x0003b8, 0x0003b8, 0x000398, 0x000398},
+ {0x0003b9, 0x0003b9, 0x000399, 0x000399},
+ {0x0003ba, 0x0003ba, 0x00039a, 0x00039a},
+ {0x0003bb, 0x0003bb, 0x00039b, 0x00039b},
+ {0x0003bc, 0x0003bc, 0x00039c, 0x00039c},
+ {0x0003bd, 0x0003bd, 0x00039d, 0x00039d},
+ {0x0003be, 0x0003be, 0x00039e, 0x00039e},
+ {0x0003bf, 0x0003bf, 0x00039f, 0x00039f},
+ {0x0003c0, 0x0003c0, 0x0003a0, 0x0003a0},
+ {0x0003c1, 0x0003c1, 0x0003a1, 0x0003a1},
+ {0x0003c2, 0x0003c2, 0x0003a3, 0x0003a3},
+ {0x0003c3, 0x0003c3, 0x0003a3, 0x0003a3},
+ {0x0003c4, 0x0003c4, 0x0003a4, 0x0003a4},
+ {0x0003c5, 0x0003c5, 0x0003a5, 0x0003a5},
+ {0x0003c6, 0x0003c6, 0x0003a6, 0x0003a6},
+ {0x0003c7, 0x0003c7, 0x0003a7, 0x0003a7},
+ {0x0003c8, 0x0003c8, 0x0003a8, 0x0003a8},
+ {0x0003c9, 0x0003c9, 0x0003a9, 0x0003a9},
+ {0x0003ca, 0x0003ca, 0x0003aa, 0x0003aa},
+ {0x0003cb, 0x0003cb, 0x0003ab, 0x0003ab},
+ {0x0003cc, 0x0003cc, 0x00038c, 0x00038c},
+ {0x0003cd, 0x0003cd, 0x00038e, 0x00038e},
+ {0x0003ce, 0x0003ce, 0x00038f, 0x00038f},
+ {0x0003cf, 0x0003d7, 0x0003cf, 0x0003cf},
+ {0x0003d0, 0x0003d0, 0x000392, 0x000392},
+ {0x0003d1, 0x0003d1, 0x000398, 0x000398},
+ {0x0003d5, 0x0003d5, 0x0003a6, 0x0003a6},
+ {0x0003d6, 0x0003d6, 0x0003a0, 0x0003a0},
+ {0x0003d7, 0x0003d7, 0x0003cf, 0x0003cf},
+ {0x0003d8, 0x0003d9, 0x0003d8, 0x0003d8},
+ {0x0003d9, 0x0003d9, 0x0003d8, 0x0003d8},
+ {0x0003da, 0x0003db, 0x0003da, 0x0003da},
+ {0x0003db, 0x0003db, 0x0003da, 0x0003da},
+ {0x0003dc, 0x0003dd, 0x0003dc, 0x0003dc},
+ {0x0003dd, 0x0003dd, 0x0003dc, 0x0003dc},
+ {0x0003de, 0x0003df, 0x0003de, 0x0003de},
+ {0x0003df, 0x0003df, 0x0003de, 0x0003de},
+ {0x0003e0, 0x0003e1, 0x0003e0, 0x0003e0},
+ {0x0003e1, 0x0003e1, 0x0003e0, 0x0003e0},
+ {0x0003e2, 0x0003e3, 0x0003e2, 0x0003e2},
+ {0x0003e3, 0x0003e3, 0x0003e2, 0x0003e2},
+ {0x0003e4, 0x0003e5, 0x0003e4, 0x0003e4},
+ {0x0003e5, 0x0003e5, 0x0003e4, 0x0003e4},
+ {0x0003e6, 0x0003e7, 0x0003e6, 0x0003e6},
+ {0x0003e7, 0x0003e7, 0x0003e6, 0x0003e6},
+ {0x0003e8, 0x0003e9, 0x0003e8, 0x0003e8},
+ {0x0003e9, 0x0003e9, 0x0003e8, 0x0003e8},
+ {0x0003ea, 0x0003eb, 0x0003ea, 0x0003ea},
+ {0x0003eb, 0x0003eb, 0x0003ea, 0x0003ea},
+ {0x0003ec, 0x0003ed, 0x0003ec, 0x0003ec},
+ {0x0003ed, 0x0003ed, 0x0003ec, 0x0003ec},
+ {0x0003ee, 0x0003ef, 0x0003ee, 0x0003ee},
+ {0x0003ef, 0x0003ef, 0x0003ee, 0x0003ee},
+ {0x0003f0, 0x0003f0, 0x00039a, 0x00039a},
+ {0x0003f1, 0x0003f1, 0x0003a1, 0x0003a1},
+ {0x0003f2, 0x0003f2, 0x0003f9, 0x0003f9},
+ {0x0003f3, 0x0003f3, 0x00037f, 0x00037f},
+ {0x0003f4, 0x0003b8, 0x0003f4, 0x0003f4},
+ {0x0003f5, 0x0003f5, 0x000395, 0x000395},
+ {0x0003f7, 0x0003f8, 0x0003f7, 0x0003f7},
+ {0x0003f8, 0x0003f8, 0x0003f7, 0x0003f7},
+ {0x0003f9, 0x0003f2, 0x0003f9, 0x0003f9},
+ {0x0003fa, 0x0003fb, 0x0003fa, 0x0003fa},
+ {0x0003fb, 0x0003fb, 0x0003fa, 0x0003fa},
+ {0x0003fd, 0x00037b, 0x0003fd, 0x0003fd},
+ {0x0003fe, 0x00037c, 0x0003fe, 0x0003fe},
+ {0x0003ff, 0x00037d, 0x0003ff, 0x0003ff},
+ {0x000400, 0x000450, 0x000400, 0x000400},
+ {0x000401, 0x000451, 0x000401, 0x000401},
+ {0x000402, 0x000452, 0x000402, 0x000402},
+ {0x000403, 0x000453, 0x000403, 0x000403},
+ {0x000404, 0x000454, 0x000404, 0x000404},
+ {0x000405, 0x000455, 0x000405, 0x000405},
+ {0x000406, 0x000456, 0x000406, 0x000406},
+ {0x000407, 0x000457, 0x000407, 0x000407},
+ {0x000408, 0x000458, 0x000408, 0x000408},
+ {0x000409, 0x000459, 0x000409, 0x000409},
+ {0x00040a, 0x00045a, 0x00040a, 0x00040a},
+ {0x00040b, 0x00045b, 0x00040b, 0x00040b},
+ {0x00040c, 0x00045c, 0x00040c, 0x00040c},
+ {0x00040d, 0x00045d, 0x00040d, 0x00040d},
+ {0x00040e, 0x00045e, 0x00040e, 0x00040e},
+ {0x00040f, 0x00045f, 0x00040f, 0x00040f},
+ {0x000410, 0x000430, 0x000410, 0x000410},
+ {0x000411, 0x000431, 0x000411, 0x000411},
+ {0x000412, 0x000432, 0x000412, 0x000412},
+ {0x000413, 0x000433, 0x000413, 0x000413},
+ {0x000414, 0x000434, 0x000414, 0x000414},
+ {0x000415, 0x000435, 0x000415, 0x000415},
+ {0x000416, 0x000436, 0x000416, 0x000416},
+ {0x000417, 0x000437, 0x000417, 0x000417},
+ {0x000418, 0x000438, 0x000418, 0x000418},
+ {0x000419, 0x000439, 0x000419, 0x000419},
+ {0x00041a, 0x00043a, 0x00041a, 0x00041a},
+ {0x00041b, 0x00043b, 0x00041b, 0x00041b},
+ {0x00041c, 0x00043c, 0x00041c, 0x00041c},
+ {0x00041d, 0x00043d, 0x00041d, 0x00041d},
+ {0x00041e, 0x00043e, 0x00041e, 0x00041e},
+ {0x00041f, 0x00043f, 0x00041f, 0x00041f},
+ {0x000420, 0x000440, 0x000420, 0x000420},
+ {0x000421, 0x000441, 0x000421, 0x000421},
+ {0x000422, 0x000442, 0x000422, 0x000422},
+ {0x000423, 0x000443, 0x000423, 0x000423},
+ {0x000424, 0x000444, 0x000424, 0x000424},
+ {0x000425, 0x000445, 0x000425, 0x000425},
+ {0x000426, 0x000446, 0x000426, 0x000426},
+ {0x000427, 0x000447, 0x000427, 0x000427},
+ {0x000428, 0x000448, 0x000428, 0x000428},
+ {0x000429, 0x000449, 0x000429, 0x000429},
+ {0x00042a, 0x00044a, 0x00042a, 0x00042a},
+ {0x00042b, 0x00044b, 0x00042b, 0x00042b},
+ {0x00042c, 0x00044c, 0x00042c, 0x00042c},
+ {0x00042d, 0x00044d, 0x00042d, 0x00042d},
+ {0x00042e, 0x00044e, 0x00042e, 0x00042e},
+ {0x00042f, 0x00044f, 0x00042f, 0x00042f},
+ {0x000430, 0x000430, 0x000410, 0x000410},
+ {0x000431, 0x000431, 0x000411, 0x000411},
+ {0x000432, 0x000432, 0x000412, 0x000412},
+ {0x000433, 0x000433, 0x000413, 0x000413},
+ {0x000434, 0x000434, 0x000414, 0x000414},
+ {0x000435, 0x000435, 0x000415, 0x000415},
+ {0x000436, 0x000436, 0x000416, 0x000416},
+ {0x000437, 0x000437, 0x000417, 0x000417},
+ {0x000438, 0x000438, 0x000418, 0x000418},
+ {0x000439, 0x000439, 0x000419, 0x000419},
+ {0x00043a, 0x00043a, 0x00041a, 0x00041a},
+ {0x00043b, 0x00043b, 0x00041b, 0x00041b},
+ {0x00043c, 0x00043c, 0x00041c, 0x00041c},
+ {0x00043d, 0x00043d, 0x00041d, 0x00041d},
+ {0x00043e, 0x00043e, 0x00041e, 0x00041e},
+ {0x00043f, 0x00043f, 0x00041f, 0x00041f},
+ {0x000440, 0x000440, 0x000420, 0x000420},
+ {0x000441, 0x000441, 0x000421, 0x000421},
+ {0x000442, 0x000442, 0x000422, 0x000422},
+ {0x000443, 0x000443, 0x000423, 0x000423},
+ {0x000444, 0x000444, 0x000424, 0x000424},
+ {0x000445, 0x000445, 0x000425, 0x000425},
+ {0x000446, 0x000446, 0x000426, 0x000426},
+ {0x000447, 0x000447, 0x000427, 0x000427},
+ {0x000448, 0x000448, 0x000428, 0x000428},
+ {0x000449, 0x000449, 0x000429, 0x000429},
+ {0x00044a, 0x00044a, 0x00042a, 0x00042a},
+ {0x00044b, 0x00044b, 0x00042b, 0x00042b},
+ {0x00044c, 0x00044c, 0x00042c, 0x00042c},
+ {0x00044d, 0x00044d, 0x00042d, 0x00042d},
+ {0x00044e, 0x00044e, 0x00042e, 0x00042e},
+ {0x00044f, 0x00044f, 0x00042f, 0x00042f},
+ {0x000450, 0x000450, 0x000400, 0x000400},
+ {0x000451, 0x000451, 0x000401, 0x000401},
+ {0x000452, 0x000452, 0x000402, 0x000402},
+ {0x000453, 0x000453, 0x000403, 0x000403},
+ {0x000454, 0x000454, 0x000404, 0x000404},
+ {0x000455, 0x000455, 0x000405, 0x000405},
+ {0x000456, 0x000456, 0x000406, 0x000406},
+ {0x000457, 0x000457, 0x000407, 0x000407},
+ {0x000458, 0x000458, 0x000408, 0x000408},
+ {0x000459, 0x000459, 0x000409, 0x000409},
+ {0x00045a, 0x00045a, 0x00040a, 0x00040a},
+ {0x00045b, 0x00045b, 0x00040b, 0x00040b},
+ {0x00045c, 0x00045c, 0x00040c, 0x00040c},
+ {0x00045d, 0x00045d, 0x00040d, 0x00040d},
+ {0x00045e, 0x00045e, 0x00040e, 0x00040e},
+ {0x00045f, 0x00045f, 0x00040f, 0x00040f},
+ {0x000460, 0x000461, 0x000460, 0x000460},
+ {0x000461, 0x000461, 0x000460, 0x000460},
+ {0x000462, 0x000463, 0x000462, 0x000462},
+ {0x000463, 0x000463, 0x000462, 0x000462},
+ {0x000464, 0x000465, 0x000464, 0x000464},
+ {0x000465, 0x000465, 0x000464, 0x000464},
+ {0x000466, 0x000467, 0x000466, 0x000466},
+ {0x000467, 0x000467, 0x000466, 0x000466},
+ {0x000468, 0x000469, 0x000468, 0x000468},
+ {0x000469, 0x000469, 0x000468, 0x000468},
+ {0x00046a, 0x00046b, 0x00046a, 0x00046a},
+ {0x00046b, 0x00046b, 0x00046a, 0x00046a},
+ {0x00046c, 0x00046d, 0x00046c, 0x00046c},
+ {0x00046d, 0x00046d, 0x00046c, 0x00046c},
+ {0x00046e, 0x00046f, 0x00046e, 0x00046e},
+ {0x00046f, 0x00046f, 0x00046e, 0x00046e},
+ {0x000470, 0x000471, 0x000470, 0x000470},
+ {0x000471, 0x000471, 0x000470, 0x000470},
+ {0x000472, 0x000473, 0x000472, 0x000472},
+ {0x000473, 0x000473, 0x000472, 0x000472},
+ {0x000474, 0x000475, 0x000474, 0x000474},
+ {0x000475, 0x000475, 0x000474, 0x000474},
+ {0x000476, 0x000477, 0x000476, 0x000476},
+ {0x000477, 0x000477, 0x000476, 0x000476},
+ {0x000478, 0x000479, 0x000478, 0x000478},
+ {0x000479, 0x000479, 0x000478, 0x000478},
+ {0x00047a, 0x00047b, 0x00047a, 0x00047a},
+ {0x00047b, 0x00047b, 0x00047a, 0x00047a},
+ {0x00047c, 0x00047d, 0x00047c, 0x00047c},
+ {0x00047d, 0x00047d, 0x00047c, 0x00047c},
+ {0x00047e, 0x00047f, 0x00047e, 0x00047e},
+ {0x00047f, 0x00047f, 0x00047e, 0x00047e},
+ {0x000480, 0x000481, 0x000480, 0x000480},
+ {0x000481, 0x000481, 0x000480, 0x000480},
+ {0x00048a, 0x00048b, 0x00048a, 0x00048a},
+ {0x00048b, 0x00048b, 0x00048a, 0x00048a},
+ {0x00048c, 0x00048d, 0x00048c, 0x00048c},
+ {0x00048d, 0x00048d, 0x00048c, 0x00048c},
+ {0x00048e, 0x00048f, 0x00048e, 0x00048e},
+ {0x00048f, 0x00048f, 0x00048e, 0x00048e},
+ {0x000490, 0x000491, 0x000490, 0x000490},
+ {0x000491, 0x000491, 0x000490, 0x000490},
+ {0x000492, 0x000493, 0x000492, 0x000492},
+ {0x000493, 0x000493, 0x000492, 0x000492},
+ {0x000494, 0x000495, 0x000494, 0x000494},
+ {0x000495, 0x000495, 0x000494, 0x000494},
+ {0x000496, 0x000497, 0x000496, 0x000496},
+ {0x000497, 0x000497, 0x000496, 0x000496},
+ {0x000498, 0x000499, 0x000498, 0x000498},
+ {0x000499, 0x000499, 0x000498, 0x000498},
+ {0x00049a, 0x00049b, 0x00049a, 0x00049a},
+ {0x00049b, 0x00049b, 0x00049a, 0x00049a},
+ {0x00049c, 0x00049d, 0x00049c, 0x00049c},
+ {0x00049d, 0x00049d, 0x00049c, 0x00049c},
+ {0x00049e, 0x00049f, 0x00049e, 0x00049e},
+ {0x00049f, 0x00049f, 0x00049e, 0x00049e},
+ {0x0004a0, 0x0004a1, 0x0004a0, 0x0004a0},
+ {0x0004a1, 0x0004a1, 0x0004a0, 0x0004a0},
+ {0x0004a2, 0x0004a3, 0x0004a2, 0x0004a2},
+ {0x0004a3, 0x0004a3, 0x0004a2, 0x0004a2},
+ {0x0004a4, 0x0004a5, 0x0004a4, 0x0004a4},
+ {0x0004a5, 0x0004a5, 0x0004a4, 0x0004a4},
+ {0x0004a6, 0x0004a7, 0x0004a6, 0x0004a6},
+ {0x0004a7, 0x0004a7, 0x0004a6, 0x0004a6},
+ {0x0004a8, 0x0004a9, 0x0004a8, 0x0004a8},
+ {0x0004a9, 0x0004a9, 0x0004a8, 0x0004a8},
+ {0x0004aa, 0x0004ab, 0x0004aa, 0x0004aa},
+ {0x0004ab, 0x0004ab, 0x0004aa, 0x0004aa},
+ {0x0004ac, 0x0004ad, 0x0004ac, 0x0004ac},
+ {0x0004ad, 0x0004ad, 0x0004ac, 0x0004ac},
+ {0x0004ae, 0x0004af, 0x0004ae, 0x0004ae},
+ {0x0004af, 0x0004af, 0x0004ae, 0x0004ae},
+ {0x0004b0, 0x0004b1, 0x0004b0, 0x0004b0},
+ {0x0004b1, 0x0004b1, 0x0004b0, 0x0004b0},
+ {0x0004b2, 0x0004b3, 0x0004b2, 0x0004b2},
+ {0x0004b3, 0x0004b3, 0x0004b2, 0x0004b2},
+ {0x0004b4, 0x0004b5, 0x0004b4, 0x0004b4},
+ {0x0004b5, 0x0004b5, 0x0004b4, 0x0004b4},
+ {0x0004b6, 0x0004b7, 0x0004b6, 0x0004b6},
+ {0x0004b7, 0x0004b7, 0x0004b6, 0x0004b6},
+ {0x0004b8, 0x0004b9, 0x0004b8, 0x0004b8},
+ {0x0004b9, 0x0004b9, 0x0004b8, 0x0004b8},
+ {0x0004ba, 0x0004bb, 0x0004ba, 0x0004ba},
+ {0x0004bb, 0x0004bb, 0x0004ba, 0x0004ba},
+ {0x0004bc, 0x0004bd, 0x0004bc, 0x0004bc},
+ {0x0004bd, 0x0004bd, 0x0004bc, 0x0004bc},
+ {0x0004be, 0x0004bf, 0x0004be, 0x0004be},
+ {0x0004bf, 0x0004bf, 0x0004be, 0x0004be},
+ {0x0004c0, 0x0004cf, 0x0004c0, 0x0004c0},
+ {0x0004c1, 0x0004c2, 0x0004c1, 0x0004c1},
+ {0x0004c2, 0x0004c2, 0x0004c1, 0x0004c1},
+ {0x0004c3, 0x0004c4, 0x0004c3, 0x0004c3},
+ {0x0004c4, 0x0004c4, 0x0004c3, 0x0004c3},
+ {0x0004c5, 0x0004c6, 0x0004c5, 0x0004c5},
+ {0x0004c6, 0x0004c6, 0x0004c5, 0x0004c5},
+ {0x0004c7, 0x0004c8, 0x0004c7, 0x0004c7},
+ {0x0004c8, 0x0004c8, 0x0004c7, 0x0004c7},
+ {0x0004c9, 0x0004ca, 0x0004c9, 0x0004c9},
+ {0x0004ca, 0x0004ca, 0x0004c9, 0x0004c9},
+ {0x0004cb, 0x0004cc, 0x0004cb, 0x0004cb},
+ {0x0004cc, 0x0004cc, 0x0004cb, 0x0004cb},
+ {0x0004cd, 0x0004ce, 0x0004cd, 0x0004cd},
+ {0x0004ce, 0x0004ce, 0x0004cd, 0x0004cd},
+ {0x0004cf, 0x0004cf, 0x0004c0, 0x0004c0},
+ {0x0004d0, 0x0004d1, 0x0004d0, 0x0004d0},
+ {0x0004d1, 0x0004d1, 0x0004d0, 0x0004d0},
+ {0x0004d2, 0x0004d3, 0x0004d2, 0x0004d2},
+ {0x0004d3, 0x0004d3, 0x0004d2, 0x0004d2},
+ {0x0004d4, 0x0004d5, 0x0004d4, 0x0004d4},
+ {0x0004d5, 0x0004d5, 0x0004d4, 0x0004d4},
+ {0x0004d6, 0x0004d7, 0x0004d6, 0x0004d6},
+ {0x0004d7, 0x0004d7, 0x0004d6, 0x0004d6},
+ {0x0004d8, 0x0004d9, 0x0004d8, 0x0004d8},
+ {0x0004d9, 0x0004d9, 0x0004d8, 0x0004d8},
+ {0x0004da, 0x0004db, 0x0004da, 0x0004da},
+ {0x0004db, 0x0004db, 0x0004da, 0x0004da},
+ {0x0004dc, 0x0004dd, 0x0004dc, 0x0004dc},
+ {0x0004dd, 0x0004dd, 0x0004dc, 0x0004dc},
+ {0x0004de, 0x0004df, 0x0004de, 0x0004de},
+ {0x0004df, 0x0004df, 0x0004de, 0x0004de},
+ {0x0004e0, 0x0004e1, 0x0004e0, 0x0004e0},
+ {0x0004e1, 0x0004e1, 0x0004e0, 0x0004e0},
+ {0x0004e2, 0x0004e3, 0x0004e2, 0x0004e2},
+ {0x0004e3, 0x0004e3, 0x0004e2, 0x0004e2},
+ {0x0004e4, 0x0004e5, 0x0004e4, 0x0004e4},
+ {0x0004e5, 0x0004e5, 0x0004e4, 0x0004e4},
+ {0x0004e6, 0x0004e7, 0x0004e6, 0x0004e6},
+ {0x0004e7, 0x0004e7, 0x0004e6, 0x0004e6},
+ {0x0004e8, 0x0004e9, 0x0004e8, 0x0004e8},
+ {0x0004e9, 0x0004e9, 0x0004e8, 0x0004e8},
+ {0x0004ea, 0x0004eb, 0x0004ea, 0x0004ea},
+ {0x0004eb, 0x0004eb, 0x0004ea, 0x0004ea},
+ {0x0004ec, 0x0004ed, 0x0004ec, 0x0004ec},
+ {0x0004ed, 0x0004ed, 0x0004ec, 0x0004ec},
+ {0x0004ee, 0x0004ef, 0x0004ee, 0x0004ee},
+ {0x0004ef, 0x0004ef, 0x0004ee, 0x0004ee},
+ {0x0004f0, 0x0004f1, 0x0004f0, 0x0004f0},
+ {0x0004f1, 0x0004f1, 0x0004f0, 0x0004f0},
+ {0x0004f2, 0x0004f3, 0x0004f2, 0x0004f2},
+ {0x0004f3, 0x0004f3, 0x0004f2, 0x0004f2},
+ {0x0004f4, 0x0004f5, 0x0004f4, 0x0004f4},
+ {0x0004f5, 0x0004f5, 0x0004f4, 0x0004f4},
+ {0x0004f6, 0x0004f7, 0x0004f6, 0x0004f6},
+ {0x0004f7, 0x0004f7, 0x0004f6, 0x0004f6},
+ {0x0004f8, 0x0004f9, 0x0004f8, 0x0004f8},
+ {0x0004f9, 0x0004f9, 0x0004f8, 0x0004f8},
+ {0x0004fa, 0x0004fb, 0x0004fa, 0x0004fa},
+ {0x0004fb, 0x0004fb, 0x0004fa, 0x0004fa},
+ {0x0004fc, 0x0004fd, 0x0004fc, 0x0004fc},
+ {0x0004fd, 0x0004fd, 0x0004fc, 0x0004fc},
+ {0x0004fe, 0x0004ff, 0x0004fe, 0x0004fe},
+ {0x0004ff, 0x0004ff, 0x0004fe, 0x0004fe},
+ {0x000500, 0x000501, 0x000500, 0x000500},
+ {0x000501, 0x000501, 0x000500, 0x000500},
+ {0x000502, 0x000503, 0x000502, 0x000502},
+ {0x000503, 0x000503, 0x000502, 0x000502},
+ {0x000504, 0x000505, 0x000504, 0x000504},
+ {0x000505, 0x000505, 0x000504, 0x000504},
+ {0x000506, 0x000507, 0x000506, 0x000506},
+ {0x000507, 0x000507, 0x000506, 0x000506},
+ {0x000508, 0x000509, 0x000508, 0x000508},
+ {0x000509, 0x000509, 0x000508, 0x000508},
+ {0x00050a, 0x00050b, 0x00050a, 0x00050a},
+ {0x00050b, 0x00050b, 0x00050a, 0x00050a},
+ {0x00050c, 0x00050d, 0x00050c, 0x00050c},
+ {0x00050d, 0x00050d, 0x00050c, 0x00050c},
+ {0x00050e, 0x00050f, 0x00050e, 0x00050e},
+ {0x00050f, 0x00050f, 0x00050e, 0x00050e},
+ {0x000510, 0x000511, 0x000510, 0x000510},
+ {0x000511, 0x000511, 0x000510, 0x000510},
+ {0x000512, 0x000513, 0x000512, 0x000512},
+ {0x000513, 0x000513, 0x000512, 0x000512},
+ {0x000514, 0x000515, 0x000514, 0x000514},
+ {0x000515, 0x000515, 0x000514, 0x000514},
+ {0x000516, 0x000517, 0x000516, 0x000516},
+ {0x000517, 0x000517, 0x000516, 0x000516},
+ {0x000518, 0x000519, 0x000518, 0x000518},
+ {0x000519, 0x000519, 0x000518, 0x000518},
+ {0x00051a, 0x00051b, 0x00051a, 0x00051a},
+ {0x00051b, 0x00051b, 0x00051a, 0x00051a},
+ {0x00051c, 0x00051d, 0x00051c, 0x00051c},
+ {0x00051d, 0x00051d, 0x00051c, 0x00051c},
+ {0x00051e, 0x00051f, 0x00051e, 0x00051e},
+ {0x00051f, 0x00051f, 0x00051e, 0x00051e},
+ {0x000520, 0x000521, 0x000520, 0x000520},
+ {0x000521, 0x000521, 0x000520, 0x000520},
+ {0x000522, 0x000523, 0x000522, 0x000522},
+ {0x000523, 0x000523, 0x000522, 0x000522},
+ {0x000524, 0x000525, 0x000524, 0x000524},
+ {0x000525, 0x000525, 0x000524, 0x000524},
+ {0x000526, 0x000527, 0x000526, 0x000526},
+ {0x000527, 0x000527, 0x000526, 0x000526},
+ {0x000528, 0x000529, 0x000528, 0x000528},
+ {0x000529, 0x000529, 0x000528, 0x000528},
+ {0x00052a, 0x00052b, 0x00052a, 0x00052a},
+ {0x00052b, 0x00052b, 0x00052a, 0x00052a},
+ {0x00052c, 0x00052d, 0x00052c, 0x00052c},
+ {0x00052d, 0x00052d, 0x00052c, 0x00052c},
+ {0x00052e, 0x00052f, 0x00052e, 0x00052e},
+ {0x00052f, 0x00052f, 0x00052e, 0x00052e},
+ {0x000531, 0x000561, 0x000531, 0x000531},
+ {0x000532, 0x000562, 0x000532, 0x000532},
+ {0x000533, 0x000563, 0x000533, 0x000533},
+ {0x000534, 0x000564, 0x000534, 0x000534},
+ {0x000535, 0x000565, 0x000535, 0x000535},
+ {0x000536, 0x000566, 0x000536, 0x000536},
+ {0x000537, 0x000567, 0x000537, 0x000537},
+ {0x000538, 0x000568, 0x000538, 0x000538},
+ {0x000539, 0x000569, 0x000539, 0x000539},
+ {0x00053a, 0x00056a, 0x00053a, 0x00053a},
+ {0x00053b, 0x00056b, 0x00053b, 0x00053b},
+ {0x00053c, 0x00056c, 0x00053c, 0x00053c},
+ {0x00053d, 0x00056d, 0x00053d, 0x00053d},
+ {0x00053e, 0x00056e, 0x00053e, 0x00053e},
+ {0x00053f, 0x00056f, 0x00053f, 0x00053f},
+ {0x000540, 0x000570, 0x000540, 0x000540},
+ {0x000541, 0x000571, 0x000541, 0x000541},
+ {0x000542, 0x000572, 0x000542, 0x000542},
+ {0x000543, 0x000573, 0x000543, 0x000543},
+ {0x000544, 0x000574, 0x000544, 0x000544},
+ {0x000545, 0x000575, 0x000545, 0x000545},
+ {0x000546, 0x000576, 0x000546, 0x000546},
+ {0x000547, 0x000577, 0x000547, 0x000547},
+ {0x000548, 0x000578, 0x000548, 0x000548},
+ {0x000549, 0x000579, 0x000549, 0x000549},
+ {0x00054a, 0x00057a, 0x00054a, 0x00054a},
+ {0x00054b, 0x00057b, 0x00054b, 0x00054b},
+ {0x00054c, 0x00057c, 0x00054c, 0x00054c},
+ {0x00054d, 0x00057d, 0x00054d, 0x00054d},
+ {0x00054e, 0x00057e, 0x00054e, 0x00054e},
+ {0x00054f, 0x00057f, 0x00054f, 0x00054f},
+ {0x000550, 0x000580, 0x000550, 0x000550},
+ {0x000551, 0x000581, 0x000551, 0x000551},
+ {0x000552, 0x000582, 0x000552, 0x000552},
+ {0x000553, 0x000583, 0x000553, 0x000553},
+ {0x000554, 0x000584, 0x000554, 0x000554},
+ {0x000555, 0x000585, 0x000555, 0x000555},
+ {0x000556, 0x000586, 0x000556, 0x000556},
+ {0x000561, 0x000561, 0x000531, 0x000531},
+ {0x000562, 0x000562, 0x000532, 0x000532},
+ {0x000563, 0x000563, 0x000533, 0x000533},
+ {0x000564, 0x000564, 0x000534, 0x000534},
+ {0x000565, 0x000565, 0x000535, 0x000535},
+ {0x000566, 0x000566, 0x000536, 0x000536},
+ {0x000567, 0x000567, 0x000537, 0x000537},
+ {0x000568, 0x000568, 0x000538, 0x000538},
+ {0x000569, 0x000569, 0x000539, 0x000539},
+ {0x00056a, 0x00056a, 0x00053a, 0x00053a},
+ {0x00056b, 0x00056b, 0x00053b, 0x00053b},
+ {0x00056c, 0x00056c, 0x00053c, 0x00053c},
+ {0x00056d, 0x00056d, 0x00053d, 0x00053d},
+ {0x00056e, 0x00056e, 0x00053e, 0x00053e},
+ {0x00056f, 0x00056f, 0x00053f, 0x00053f},
+ {0x000570, 0x000570, 0x000540, 0x000540},
+ {0x000571, 0x000571, 0x000541, 0x000541},
+ {0x000572, 0x000572, 0x000542, 0x000542},
+ {0x000573, 0x000573, 0x000543, 0x000543},
+ {0x000574, 0x000574, 0x000544, 0x000544},
+ {0x000575, 0x000575, 0x000545, 0x000545},
+ {0x000576, 0x000576, 0x000546, 0x000546},
+ {0x000577, 0x000577, 0x000547, 0x000547},
+ {0x000578, 0x000578, 0x000548, 0x000548},
+ {0x000579, 0x000579, 0x000549, 0x000549},
+ {0x00057a, 0x00057a, 0x00054a, 0x00054a},
+ {0x00057b, 0x00057b, 0x00054b, 0x00054b},
+ {0x00057c, 0x00057c, 0x00054c, 0x00054c},
+ {0x00057d, 0x00057d, 0x00054d, 0x00054d},
+ {0x00057e, 0x00057e, 0x00054e, 0x00054e},
+ {0x00057f, 0x00057f, 0x00054f, 0x00054f},
+ {0x000580, 0x000580, 0x000550, 0x000550},
+ {0x000581, 0x000581, 0x000551, 0x000551},
+ {0x000582, 0x000582, 0x000552, 0x000552},
+ {0x000583, 0x000583, 0x000553, 0x000553},
+ {0x000584, 0x000584, 0x000554, 0x000554},
+ {0x000585, 0x000585, 0x000555, 0x000555},
+ {0x000586, 0x000586, 0x000556, 0x000556},
+ {0x0010a0, 0x002d00, 0x0010a0, 0x0010a0},
+ {0x0010a1, 0x002d01, 0x0010a1, 0x0010a1},
+ {0x0010a2, 0x002d02, 0x0010a2, 0x0010a2},
+ {0x0010a3, 0x002d03, 0x0010a3, 0x0010a3},
+ {0x0010a4, 0x002d04, 0x0010a4, 0x0010a4},
+ {0x0010a5, 0x002d05, 0x0010a5, 0x0010a5},
+ {0x0010a6, 0x002d06, 0x0010a6, 0x0010a6},
+ {0x0010a7, 0x002d07, 0x0010a7, 0x0010a7},
+ {0x0010a8, 0x002d08, 0x0010a8, 0x0010a8},
+ {0x0010a9, 0x002d09, 0x0010a9, 0x0010a9},
+ {0x0010aa, 0x002d0a, 0x0010aa, 0x0010aa},
+ {0x0010ab, 0x002d0b, 0x0010ab, 0x0010ab},
+ {0x0010ac, 0x002d0c, 0x0010ac, 0x0010ac},
+ {0x0010ad, 0x002d0d, 0x0010ad, 0x0010ad},
+ {0x0010ae, 0x002d0e, 0x0010ae, 0x0010ae},
+ {0x0010af, 0x002d0f, 0x0010af, 0x0010af},
+ {0x0010b0, 0x002d10, 0x0010b0, 0x0010b0},
+ {0x0010b1, 0x002d11, 0x0010b1, 0x0010b1},
+ {0x0010b2, 0x002d12, 0x0010b2, 0x0010b2},
+ {0x0010b3, 0x002d13, 0x0010b3, 0x0010b3},
+ {0x0010b4, 0x002d14, 0x0010b4, 0x0010b4},
+ {0x0010b5, 0x002d15, 0x0010b5, 0x0010b5},
+ {0x0010b6, 0x002d16, 0x0010b6, 0x0010b6},
+ {0x0010b7, 0x002d17, 0x0010b7, 0x0010b7},
+ {0x0010b8, 0x002d18, 0x0010b8, 0x0010b8},
+ {0x0010b9, 0x002d19, 0x0010b9, 0x0010b9},
+ {0x0010ba, 0x002d1a, 0x0010ba, 0x0010ba},
+ {0x0010bb, 0x002d1b, 0x0010bb, 0x0010bb},
+ {0x0010bc, 0x002d1c, 0x0010bc, 0x0010bc},
+ {0x0010bd, 0x002d1d, 0x0010bd, 0x0010bd},
+ {0x0010be, 0x002d1e, 0x0010be, 0x0010be},
+ {0x0010bf, 0x002d1f, 0x0010bf, 0x0010bf},
+ {0x0010c0, 0x002d20, 0x0010c0, 0x0010c0},
+ {0x0010c1, 0x002d21, 0x0010c1, 0x0010c1},
+ {0x0010c2, 0x002d22, 0x0010c2, 0x0010c2},
+ {0x0010c3, 0x002d23, 0x0010c3, 0x0010c3},
+ {0x0010c4, 0x002d24, 0x0010c4, 0x0010c4},
+ {0x0010c5, 0x002d25, 0x0010c5, 0x0010c5},
+ {0x0010c7, 0x002d27, 0x0010c7, 0x0010c7},
+ {0x0010cd, 0x002d2d, 0x0010cd, 0x0010cd},
+ {0x0010d0, 0x0010d0, 0x0010d0, 0x001c90},
+ {0x0010d1, 0x0010d1, 0x0010d1, 0x001c91},
+ {0x0010d2, 0x0010d2, 0x0010d2, 0x001c92},
+ {0x0010d3, 0x0010d3, 0x0010d3, 0x001c93},
+ {0x0010d4, 0x0010d4, 0x0010d4, 0x001c94},
+ {0x0010d5, 0x0010d5, 0x0010d5, 0x001c95},
+ {0x0010d6, 0x0010d6, 0x0010d6, 0x001c96},
+ {0x0010d7, 0x0010d7, 0x0010d7, 0x001c97},
+ {0x0010d8, 0x0010d8, 0x0010d8, 0x001c98},
+ {0x0010d9, 0x0010d9, 0x0010d9, 0x001c99},
+ {0x0010da, 0x0010da, 0x0010da, 0x001c9a},
+ {0x0010db, 0x0010db, 0x0010db, 0x001c9b},
+ {0x0010dc, 0x0010dc, 0x0010dc, 0x001c9c},
+ {0x0010dd, 0x0010dd, 0x0010dd, 0x001c9d},
+ {0x0010de, 0x0010de, 0x0010de, 0x001c9e},
+ {0x0010df, 0x0010df, 0x0010df, 0x001c9f},
+ {0x0010e0, 0x0010e0, 0x0010e0, 0x001ca0},
+ {0x0010e1, 0x0010e1, 0x0010e1, 0x001ca1},
+ {0x0010e2, 0x0010e2, 0x0010e2, 0x001ca2},
+ {0x0010e3, 0x0010e3, 0x0010e3, 0x001ca3},
+ {0x0010e4, 0x0010e4, 0x0010e4, 0x001ca4},
+ {0x0010e5, 0x0010e5, 0x0010e5, 0x001ca5},
+ {0x0010e6, 0x0010e6, 0x0010e6, 0x001ca6},
+ {0x0010e7, 0x0010e7, 0x0010e7, 0x001ca7},
+ {0x0010e8, 0x0010e8, 0x0010e8, 0x001ca8},
+ {0x0010e9, 0x0010e9, 0x0010e9, 0x001ca9},
+ {0x0010ea, 0x0010ea, 0x0010ea, 0x001caa},
+ {0x0010eb, 0x0010eb, 0x0010eb, 0x001cab},
+ {0x0010ec, 0x0010ec, 0x0010ec, 0x001cac},
+ {0x0010ed, 0x0010ed, 0x0010ed, 0x001cad},
+ {0x0010ee, 0x0010ee, 0x0010ee, 0x001cae},
+ {0x0010ef, 0x0010ef, 0x0010ef, 0x001caf},
+ {0x0010f0, 0x0010f0, 0x0010f0, 0x001cb0},
+ {0x0010f1, 0x0010f1, 0x0010f1, 0x001cb1},
+ {0x0010f2, 0x0010f2, 0x0010f2, 0x001cb2},
+ {0x0010f3, 0x0010f3, 0x0010f3, 0x001cb3},
+ {0x0010f4, 0x0010f4, 0x0010f4, 0x001cb4},
+ {0x0010f5, 0x0010f5, 0x0010f5, 0x001cb5},
+ {0x0010f6, 0x0010f6, 0x0010f6, 0x001cb6},
+ {0x0010f7, 0x0010f7, 0x0010f7, 0x001cb7},
+ {0x0010f8, 0x0010f8, 0x0010f8, 0x001cb8},
+ {0x0010f9, 0x0010f9, 0x0010f9, 0x001cb9},
+ {0x0010fa, 0x0010fa, 0x0010fa, 0x001cba},
+ {0x0010fd, 0x0010fd, 0x0010fd, 0x001cbd},
+ {0x0010fe, 0x0010fe, 0x0010fe, 0x001cbe},
+ {0x0010ff, 0x0010ff, 0x0010ff, 0x001cbf},
+ {0x0013a0, 0x00ab70, 0x0013a0, 0x0013a0},
+ {0x0013a1, 0x00ab71, 0x0013a1, 0x0013a1},
+ {0x0013a2, 0x00ab72, 0x0013a2, 0x0013a2},
+ {0x0013a3, 0x00ab73, 0x0013a3, 0x0013a3},
+ {0x0013a4, 0x00ab74, 0x0013a4, 0x0013a4},
+ {0x0013a5, 0x00ab75, 0x0013a5, 0x0013a5},
+ {0x0013a6, 0x00ab76, 0x0013a6, 0x0013a6},
+ {0x0013a7, 0x00ab77, 0x0013a7, 0x0013a7},
+ {0x0013a8, 0x00ab78, 0x0013a8, 0x0013a8},
+ {0x0013a9, 0x00ab79, 0x0013a9, 0x0013a9},
+ {0x0013aa, 0x00ab7a, 0x0013aa, 0x0013aa},
+ {0x0013ab, 0x00ab7b, 0x0013ab, 0x0013ab},
+ {0x0013ac, 0x00ab7c, 0x0013ac, 0x0013ac},
+ {0x0013ad, 0x00ab7d, 0x0013ad, 0x0013ad},
+ {0x0013ae, 0x00ab7e, 0x0013ae, 0x0013ae},
+ {0x0013af, 0x00ab7f, 0x0013af, 0x0013af},
+ {0x0013b0, 0x00ab80, 0x0013b0, 0x0013b0},
+ {0x0013b1, 0x00ab81, 0x0013b1, 0x0013b1},
+ {0x0013b2, 0x00ab82, 0x0013b2, 0x0013b2},
+ {0x0013b3, 0x00ab83, 0x0013b3, 0x0013b3},
+ {0x0013b4, 0x00ab84, 0x0013b4, 0x0013b4},
+ {0x0013b5, 0x00ab85, 0x0013b5, 0x0013b5},
+ {0x0013b6, 0x00ab86, 0x0013b6, 0x0013b6},
+ {0x0013b7, 0x00ab87, 0x0013b7, 0x0013b7},
+ {0x0013b8, 0x00ab88, 0x0013b8, 0x0013b8},
+ {0x0013b9, 0x00ab89, 0x0013b9, 0x0013b9},
+ {0x0013ba, 0x00ab8a, 0x0013ba, 0x0013ba},
+ {0x0013bb, 0x00ab8b, 0x0013bb, 0x0013bb},
+ {0x0013bc, 0x00ab8c, 0x0013bc, 0x0013bc},
+ {0x0013bd, 0x00ab8d, 0x0013bd, 0x0013bd},
+ {0x0013be, 0x00ab8e, 0x0013be, 0x0013be},
+ {0x0013bf, 0x00ab8f, 0x0013bf, 0x0013bf},
+ {0x0013c0, 0x00ab90, 0x0013c0, 0x0013c0},
+ {0x0013c1, 0x00ab91, 0x0013c1, 0x0013c1},
+ {0x0013c2, 0x00ab92, 0x0013c2, 0x0013c2},
+ {0x0013c3, 0x00ab93, 0x0013c3, 0x0013c3},
+ {0x0013c4, 0x00ab94, 0x0013c4, 0x0013c4},
+ {0x0013c5, 0x00ab95, 0x0013c5, 0x0013c5},
+ {0x0013c6, 0x00ab96, 0x0013c6, 0x0013c6},
+ {0x0013c7, 0x00ab97, 0x0013c7, 0x0013c7},
+ {0x0013c8, 0x00ab98, 0x0013c8, 0x0013c8},
+ {0x0013c9, 0x00ab99, 0x0013c9, 0x0013c9},
+ {0x0013ca, 0x00ab9a, 0x0013ca, 0x0013ca},
+ {0x0013cb, 0x00ab9b, 0x0013cb, 0x0013cb},
+ {0x0013cc, 0x00ab9c, 0x0013cc, 0x0013cc},
+ {0x0013cd, 0x00ab9d, 0x0013cd, 0x0013cd},
+ {0x0013ce, 0x00ab9e, 0x0013ce, 0x0013ce},
+ {0x0013cf, 0x00ab9f, 0x0013cf, 0x0013cf},
+ {0x0013d0, 0x00aba0, 0x0013d0, 0x0013d0},
+ {0x0013d1, 0x00aba1, 0x0013d1, 0x0013d1},
+ {0x0013d2, 0x00aba2, 0x0013d2, 0x0013d2},
+ {0x0013d3, 0x00aba3, 0x0013d3, 0x0013d3},
+ {0x0013d4, 0x00aba4, 0x0013d4, 0x0013d4},
+ {0x0013d5, 0x00aba5, 0x0013d5, 0x0013d5},
+ {0x0013d6, 0x00aba6, 0x0013d6, 0x0013d6},
+ {0x0013d7, 0x00aba7, 0x0013d7, 0x0013d7},
+ {0x0013d8, 0x00aba8, 0x0013d8, 0x0013d8},
+ {0x0013d9, 0x00aba9, 0x0013d9, 0x0013d9},
+ {0x0013da, 0x00abaa, 0x0013da, 0x0013da},
+ {0x0013db, 0x00abab, 0x0013db, 0x0013db},
+ {0x0013dc, 0x00abac, 0x0013dc, 0x0013dc},
+ {0x0013dd, 0x00abad, 0x0013dd, 0x0013dd},
+ {0x0013de, 0x00abae, 0x0013de, 0x0013de},
+ {0x0013df, 0x00abaf, 0x0013df, 0x0013df},
+ {0x0013e0, 0x00abb0, 0x0013e0, 0x0013e0},
+ {0x0013e1, 0x00abb1, 0x0013e1, 0x0013e1},
+ {0x0013e2, 0x00abb2, 0x0013e2, 0x0013e2},
+ {0x0013e3, 0x00abb3, 0x0013e3, 0x0013e3},
+ {0x0013e4, 0x00abb4, 0x0013e4, 0x0013e4},
+ {0x0013e5, 0x00abb5, 0x0013e5, 0x0013e5},
+ {0x0013e6, 0x00abb6, 0x0013e6, 0x0013e6},
+ {0x0013e7, 0x00abb7, 0x0013e7, 0x0013e7},
+ {0x0013e8, 0x00abb8, 0x0013e8, 0x0013e8},
+ {0x0013e9, 0x00abb9, 0x0013e9, 0x0013e9},
+ {0x0013ea, 0x00abba, 0x0013ea, 0x0013ea},
+ {0x0013eb, 0x00abbb, 0x0013eb, 0x0013eb},
+ {0x0013ec, 0x00abbc, 0x0013ec, 0x0013ec},
+ {0x0013ed, 0x00abbd, 0x0013ed, 0x0013ed},
+ {0x0013ee, 0x00abbe, 0x0013ee, 0x0013ee},
+ {0x0013ef, 0x00abbf, 0x0013ef, 0x0013ef},
+ {0x0013f0, 0x0013f8, 0x0013f0, 0x0013f0},
+ {0x0013f1, 0x0013f9, 0x0013f1, 0x0013f1},
+ {0x0013f2, 0x0013fa, 0x0013f2, 0x0013f2},
+ {0x0013f3, 0x0013fb, 0x0013f3, 0x0013f3},
+ {0x0013f4, 0x0013fc, 0x0013f4, 0x0013f4},
+ {0x0013f5, 0x0013fd, 0x0013f5, 0x0013f5},
+ {0x0013f8, 0x0013f8, 0x0013f0, 0x0013f0},
+ {0x0013f9, 0x0013f9, 0x0013f1, 0x0013f1},
+ {0x0013fa, 0x0013fa, 0x0013f2, 0x0013f2},
+ {0x0013fb, 0x0013fb, 0x0013f3, 0x0013f3},
+ {0x0013fc, 0x0013fc, 0x0013f4, 0x0013f4},
+ {0x0013fd, 0x0013fd, 0x0013f5, 0x0013f5},
+ {0x001c80, 0x001c80, 0x000412, 0x000412},
+ {0x001c81, 0x001c81, 0x000414, 0x000414},
+ {0x001c82, 0x001c82, 0x00041e, 0x00041e},
+ {0x001c83, 0x001c83, 0x000421, 0x000421},
+ {0x001c84, 0x001c84, 0x000422, 0x000422},
+ {0x001c85, 0x001c85, 0x000422, 0x000422},
+ {0x001c86, 0x001c86, 0x00042a, 0x00042a},
+ {0x001c87, 0x001c87, 0x000462, 0x000462},
+ {0x001c88, 0x001c88, 0x00a64a, 0x00a64a},
+ {0x001c90, 0x0010d0, 0x001c90, 0x001c90},
+ {0x001c91, 0x0010d1, 0x001c91, 0x001c91},
+ {0x001c92, 0x0010d2, 0x001c92, 0x001c92},
+ {0x001c93, 0x0010d3, 0x001c93, 0x001c93},
+ {0x001c94, 0x0010d4, 0x001c94, 0x001c94},
+ {0x001c95, 0x0010d5, 0x001c95, 0x001c95},
+ {0x001c96, 0x0010d6, 0x001c96, 0x001c96},
+ {0x001c97, 0x0010d7, 0x001c97, 0x001c97},
+ {0x001c98, 0x0010d8, 0x001c98, 0x001c98},
+ {0x001c99, 0x0010d9, 0x001c99, 0x001c99},
+ {0x001c9a, 0x0010da, 0x001c9a, 0x001c9a},
+ {0x001c9b, 0x0010db, 0x001c9b, 0x001c9b},
+ {0x001c9c, 0x0010dc, 0x001c9c, 0x001c9c},
+ {0x001c9d, 0x0010dd, 0x001c9d, 0x001c9d},
+ {0x001c9e, 0x0010de, 0x001c9e, 0x001c9e},
+ {0x001c9f, 0x0010df, 0x001c9f, 0x001c9f},
+ {0x001ca0, 0x0010e0, 0x001ca0, 0x001ca0},
+ {0x001ca1, 0x0010e1, 0x001ca1, 0x001ca1},
+ {0x001ca2, 0x0010e2, 0x001ca2, 0x001ca2},
+ {0x001ca3, 0x0010e3, 0x001ca3, 0x001ca3},
+ {0x001ca4, 0x0010e4, 0x001ca4, 0x001ca4},
+ {0x001ca5, 0x0010e5, 0x001ca5, 0x001ca5},
+ {0x001ca6, 0x0010e6, 0x001ca6, 0x001ca6},
+ {0x001ca7, 0x0010e7, 0x001ca7, 0x001ca7},
+ {0x001ca8, 0x0010e8, 0x001ca8, 0x001ca8},
+ {0x001ca9, 0x0010e9, 0x001ca9, 0x001ca9},
+ {0x001caa, 0x0010ea, 0x001caa, 0x001caa},
+ {0x001cab, 0x0010eb, 0x001cab, 0x001cab},
+ {0x001cac, 0x0010ec, 0x001cac, 0x001cac},
+ {0x001cad, 0x0010ed, 0x001cad, 0x001cad},
+ {0x001cae, 0x0010ee, 0x001cae, 0x001cae},
+ {0x001caf, 0x0010ef, 0x001caf, 0x001caf},
+ {0x001cb0, 0x0010f0, 0x001cb0, 0x001cb0},
+ {0x001cb1, 0x0010f1, 0x001cb1, 0x001cb1},
+ {0x001cb2, 0x0010f2, 0x001cb2, 0x001cb2},
+ {0x001cb3, 0x0010f3, 0x001cb3, 0x001cb3},
+ {0x001cb4, 0x0010f4, 0x001cb4, 0x001cb4},
+ {0x001cb5, 0x0010f5, 0x001cb5, 0x001cb5},
+ {0x001cb6, 0x0010f6, 0x001cb6, 0x001cb6},
+ {0x001cb7, 0x0010f7, 0x001cb7, 0x001cb7},
+ {0x001cb8, 0x0010f8, 0x001cb8, 0x001cb8},
+ {0x001cb9, 0x0010f9, 0x001cb9, 0x001cb9},
+ {0x001cba, 0x0010fa, 0x001cba, 0x001cba},
+ {0x001cbd, 0x0010fd, 0x001cbd, 0x001cbd},
+ {0x001cbe, 0x0010fe, 0x001cbe, 0x001cbe},
+ {0x001cbf, 0x0010ff, 0x001cbf, 0x001cbf},
+ {0x001d79, 0x001d79, 0x00a77d, 0x00a77d},
+ {0x001d7d, 0x001d7d, 0x002c63, 0x002c63},
+ {0x001d8e, 0x001d8e, 0x00a7c6, 0x00a7c6},
+ {0x001e00, 0x001e01, 0x001e00, 0x001e00},
+ {0x001e01, 0x001e01, 0x001e00, 0x001e00},
+ {0x001e02, 0x001e03, 0x001e02, 0x001e02},
+ {0x001e03, 0x001e03, 0x001e02, 0x001e02},
+ {0x001e04, 0x001e05, 0x001e04, 0x001e04},
+ {0x001e05, 0x001e05, 0x001e04, 0x001e04},
+ {0x001e06, 0x001e07, 0x001e06, 0x001e06},
+ {0x001e07, 0x001e07, 0x001e06, 0x001e06},
+ {0x001e08, 0x001e09, 0x001e08, 0x001e08},
+ {0x001e09, 0x001e09, 0x001e08, 0x001e08},
+ {0x001e0a, 0x001e0b, 0x001e0a, 0x001e0a},
+ {0x001e0b, 0x001e0b, 0x001e0a, 0x001e0a},
+ {0x001e0c, 0x001e0d, 0x001e0c, 0x001e0c},
+ {0x001e0d, 0x001e0d, 0x001e0c, 0x001e0c},
+ {0x001e0e, 0x001e0f, 0x001e0e, 0x001e0e},
+ {0x001e0f, 0x001e0f, 0x001e0e, 0x001e0e},
+ {0x001e10, 0x001e11, 0x001e10, 0x001e10},
+ {0x001e11, 0x001e11, 0x001e10, 0x001e10},
+ {0x001e12, 0x001e13, 0x001e12, 0x001e12},
+ {0x001e13, 0x001e13, 0x001e12, 0x001e12},
+ {0x001e14, 0x001e15, 0x001e14, 0x001e14},
+ {0x001e15, 0x001e15, 0x001e14, 0x001e14},
+ {0x001e16, 0x001e17, 0x001e16, 0x001e16},
+ {0x001e17, 0x001e17, 0x001e16, 0x001e16},
+ {0x001e18, 0x001e19, 0x001e18, 0x001e18},
+ {0x001e19, 0x001e19, 0x001e18, 0x001e18},
+ {0x001e1a, 0x001e1b, 0x001e1a, 0x001e1a},
+ {0x001e1b, 0x001e1b, 0x001e1a, 0x001e1a},
+ {0x001e1c, 0x001e1d, 0x001e1c, 0x001e1c},
+ {0x001e1d, 0x001e1d, 0x001e1c, 0x001e1c},
+ {0x001e1e, 0x001e1f, 0x001e1e, 0x001e1e},
+ {0x001e1f, 0x001e1f, 0x001e1e, 0x001e1e},
+ {0x001e20, 0x001e21, 0x001e20, 0x001e20},
+ {0x001e21, 0x001e21, 0x001e20, 0x001e20},
+ {0x001e22, 0x001e23, 0x001e22, 0x001e22},
+ {0x001e23, 0x001e23, 0x001e22, 0x001e22},
+ {0x001e24, 0x001e25, 0x001e24, 0x001e24},
+ {0x001e25, 0x001e25, 0x001e24, 0x001e24},
+ {0x001e26, 0x001e27, 0x001e26, 0x001e26},
+ {0x001e27, 0x001e27, 0x001e26, 0x001e26},
+ {0x001e28, 0x001e29, 0x001e28, 0x001e28},
+ {0x001e29, 0x001e29, 0x001e28, 0x001e28},
+ {0x001e2a, 0x001e2b, 0x001e2a, 0x001e2a},
+ {0x001e2b, 0x001e2b, 0x001e2a, 0x001e2a},
+ {0x001e2c, 0x001e2d, 0x001e2c, 0x001e2c},
+ {0x001e2d, 0x001e2d, 0x001e2c, 0x001e2c},
+ {0x001e2e, 0x001e2f, 0x001e2e, 0x001e2e},
+ {0x001e2f, 0x001e2f, 0x001e2e, 0x001e2e},
+ {0x001e30, 0x001e31, 0x001e30, 0x001e30},
+ {0x001e31, 0x001e31, 0x001e30, 0x001e30},
+ {0x001e32, 0x001e33, 0x001e32, 0x001e32},
+ {0x001e33, 0x001e33, 0x001e32, 0x001e32},
+ {0x001e34, 0x001e35, 0x001e34, 0x001e34},
+ {0x001e35, 0x001e35, 0x001e34, 0x001e34},
+ {0x001e36, 0x001e37, 0x001e36, 0x001e36},
+ {0x001e37, 0x001e37, 0x001e36, 0x001e36},
+ {0x001e38, 0x001e39, 0x001e38, 0x001e38},
+ {0x001e39, 0x001e39, 0x001e38, 0x001e38},
+ {0x001e3a, 0x001e3b, 0x001e3a, 0x001e3a},
+ {0x001e3b, 0x001e3b, 0x001e3a, 0x001e3a},
+ {0x001e3c, 0x001e3d, 0x001e3c, 0x001e3c},
+ {0x001e3d, 0x001e3d, 0x001e3c, 0x001e3c},
+ {0x001e3e, 0x001e3f, 0x001e3e, 0x001e3e},
+ {0x001e3f, 0x001e3f, 0x001e3e, 0x001e3e},
+ {0x001e40, 0x001e41, 0x001e40, 0x001e40},
+ {0x001e41, 0x001e41, 0x001e40, 0x001e40},
+ {0x001e42, 0x001e43, 0x001e42, 0x001e42},
+ {0x001e43, 0x001e43, 0x001e42, 0x001e42},
+ {0x001e44, 0x001e45, 0x001e44, 0x001e44},
+ {0x001e45, 0x001e45, 0x001e44, 0x001e44},
+ {0x001e46, 0x001e47, 0x001e46, 0x001e46},
+ {0x001e47, 0x001e47, 0x001e46, 0x001e46},
+ {0x001e48, 0x001e49, 0x001e48, 0x001e48},
+ {0x001e49, 0x001e49, 0x001e48, 0x001e48},
+ {0x001e4a, 0x001e4b, 0x001e4a, 0x001e4a},
+ {0x001e4b, 0x001e4b, 0x001e4a, 0x001e4a},
+ {0x001e4c, 0x001e4d, 0x001e4c, 0x001e4c},
+ {0x001e4d, 0x001e4d, 0x001e4c, 0x001e4c},
+ {0x001e4e, 0x001e4f, 0x001e4e, 0x001e4e},
+ {0x001e4f, 0x001e4f, 0x001e4e, 0x001e4e},
+ {0x001e50, 0x001e51, 0x001e50, 0x001e50},
+ {0x001e51, 0x001e51, 0x001e50, 0x001e50},
+ {0x001e52, 0x001e53, 0x001e52, 0x001e52},
+ {0x001e53, 0x001e53, 0x001e52, 0x001e52},
+ {0x001e54, 0x001e55, 0x001e54, 0x001e54},
+ {0x001e55, 0x001e55, 0x001e54, 0x001e54},
+ {0x001e56, 0x001e57, 0x001e56, 0x001e56},
+ {0x001e57, 0x001e57, 0x001e56, 0x001e56},
+ {0x001e58, 0x001e59, 0x001e58, 0x001e58},
+ {0x001e59, 0x001e59, 0x001e58, 0x001e58},
+ {0x001e5a, 0x001e5b, 0x001e5a, 0x001e5a},
+ {0x001e5b, 0x001e5b, 0x001e5a, 0x001e5a},
+ {0x001e5c, 0x001e5d, 0x001e5c, 0x001e5c},
+ {0x001e5d, 0x001e5d, 0x001e5c, 0x001e5c},
+ {0x001e5e, 0x001e5f, 0x001e5e, 0x001e5e},
+ {0x001e5f, 0x001e5f, 0x001e5e, 0x001e5e},
+ {0x001e60, 0x001e61, 0x001e60, 0x001e60},
+ {0x001e61, 0x001e61, 0x001e60, 0x001e60},
+ {0x001e62, 0x001e63, 0x001e62, 0x001e62},
+ {0x001e63, 0x001e63, 0x001e62, 0x001e62},
+ {0x001e64, 0x001e65, 0x001e64, 0x001e64},
+ {0x001e65, 0x001e65, 0x001e64, 0x001e64},
+ {0x001e66, 0x001e67, 0x001e66, 0x001e66},
+ {0x001e67, 0x001e67, 0x001e66, 0x001e66},
+ {0x001e68, 0x001e69, 0x001e68, 0x001e68},
+ {0x001e69, 0x001e69, 0x001e68, 0x001e68},
+ {0x001e6a, 0x001e6b, 0x001e6a, 0x001e6a},
+ {0x001e6b, 0x001e6b, 0x001e6a, 0x001e6a},
+ {0x001e6c, 0x001e6d, 0x001e6c, 0x001e6c},
+ {0x001e6d, 0x001e6d, 0x001e6c, 0x001e6c},
+ {0x001e6e, 0x001e6f, 0x001e6e, 0x001e6e},
+ {0x001e6f, 0x001e6f, 0x001e6e, 0x001e6e},
+ {0x001e70, 0x001e71, 0x001e70, 0x001e70},
+ {0x001e71, 0x001e71, 0x001e70, 0x001e70},
+ {0x001e72, 0x001e73, 0x001e72, 0x001e72},
+ {0x001e73, 0x001e73, 0x001e72, 0x001e72},
+ {0x001e74, 0x001e75, 0x001e74, 0x001e74},
+ {0x001e75, 0x001e75, 0x001e74, 0x001e74},
+ {0x001e76, 0x001e77, 0x001e76, 0x001e76},
+ {0x001e77, 0x001e77, 0x001e76, 0x001e76},
+ {0x001e78, 0x001e79, 0x001e78, 0x001e78},
+ {0x001e79, 0x001e79, 0x001e78, 0x001e78},
+ {0x001e7a, 0x001e7b, 0x001e7a, 0x001e7a},
+ {0x001e7b, 0x001e7b, 0x001e7a, 0x001e7a},
+ {0x001e7c, 0x001e7d, 0x001e7c, 0x001e7c},
+ {0x001e7d, 0x001e7d, 0x001e7c, 0x001e7c},
+ {0x001e7e, 0x001e7f, 0x001e7e, 0x001e7e},
+ {0x001e7f, 0x001e7f, 0x001e7e, 0x001e7e},
+ {0x001e80, 0x001e81, 0x001e80, 0x001e80},
+ {0x001e81, 0x001e81, 0x001e80, 0x001e80},
+ {0x001e82, 0x001e83, 0x001e82, 0x001e82},
+ {0x001e83, 0x001e83, 0x001e82, 0x001e82},
+ {0x001e84, 0x001e85, 0x001e84, 0x001e84},
+ {0x001e85, 0x001e85, 0x001e84, 0x001e84},
+ {0x001e86, 0x001e87, 0x001e86, 0x001e86},
+ {0x001e87, 0x001e87, 0x001e86, 0x001e86},
+ {0x001e88, 0x001e89, 0x001e88, 0x001e88},
+ {0x001e89, 0x001e89, 0x001e88, 0x001e88},
+ {0x001e8a, 0x001e8b, 0x001e8a, 0x001e8a},
+ {0x001e8b, 0x001e8b, 0x001e8a, 0x001e8a},
+ {0x001e8c, 0x001e8d, 0x001e8c, 0x001e8c},
+ {0x001e8d, 0x001e8d, 0x001e8c, 0x001e8c},
+ {0x001e8e, 0x001e8f, 0x001e8e, 0x001e8e},
+ {0x001e8f, 0x001e8f, 0x001e8e, 0x001e8e},
+ {0x001e90, 0x001e91, 0x001e90, 0x001e90},
+ {0x001e91, 0x001e91, 0x001e90, 0x001e90},
+ {0x001e92, 0x001e93, 0x001e92, 0x001e92},
+ {0x001e93, 0x001e93, 0x001e92, 0x001e92},
+ {0x001e94, 0x001e95, 0x001e94, 0x001e94},
+ {0x001e95, 0x001e95, 0x001e94, 0x001e94},
+ {0x001e9b, 0x001e9b, 0x001e60, 0x001e60},
+ {0x001e9e, 0x0000df, 0x001e9e, 0x001e9e},
+ {0x001ea0, 0x001ea1, 0x001ea0, 0x001ea0},
+ {0x001ea1, 0x001ea1, 0x001ea0, 0x001ea0},
+ {0x001ea2, 0x001ea3, 0x001ea2, 0x001ea2},
+ {0x001ea3, 0x001ea3, 0x001ea2, 0x001ea2},
+ {0x001ea4, 0x001ea5, 0x001ea4, 0x001ea4},
+ {0x001ea5, 0x001ea5, 0x001ea4, 0x001ea4},
+ {0x001ea6, 0x001ea7, 0x001ea6, 0x001ea6},
+ {0x001ea7, 0x001ea7, 0x001ea6, 0x001ea6},
+ {0x001ea8, 0x001ea9, 0x001ea8, 0x001ea8},
+ {0x001ea9, 0x001ea9, 0x001ea8, 0x001ea8},
+ {0x001eaa, 0x001eab, 0x001eaa, 0x001eaa},
+ {0x001eab, 0x001eab, 0x001eaa, 0x001eaa},
+ {0x001eac, 0x001ead, 0x001eac, 0x001eac},
+ {0x001ead, 0x001ead, 0x001eac, 0x001eac},
+ {0x001eae, 0x001eaf, 0x001eae, 0x001eae},
+ {0x001eaf, 0x001eaf, 0x001eae, 0x001eae},
+ {0x001eb0, 0x001eb1, 0x001eb0, 0x001eb0},
+ {0x001eb1, 0x001eb1, 0x001eb0, 0x001eb0},
+ {0x001eb2, 0x001eb3, 0x001eb2, 0x001eb2},
+ {0x001eb3, 0x001eb3, 0x001eb2, 0x001eb2},
+ {0x001eb4, 0x001eb5, 0x001eb4, 0x001eb4},
+ {0x001eb5, 0x001eb5, 0x001eb4, 0x001eb4},
+ {0x001eb6, 0x001eb7, 0x001eb6, 0x001eb6},
+ {0x001eb7, 0x001eb7, 0x001eb6, 0x001eb6},
+ {0x001eb8, 0x001eb9, 0x001eb8, 0x001eb8},
+ {0x001eb9, 0x001eb9, 0x001eb8, 0x001eb8},
+ {0x001eba, 0x001ebb, 0x001eba, 0x001eba},
+ {0x001ebb, 0x001ebb, 0x001eba, 0x001eba},
+ {0x001ebc, 0x001ebd, 0x001ebc, 0x001ebc},
+ {0x001ebd, 0x001ebd, 0x001ebc, 0x001ebc},
+ {0x001ebe, 0x001ebf, 0x001ebe, 0x001ebe},
+ {0x001ebf, 0x001ebf, 0x001ebe, 0x001ebe},
+ {0x001ec0, 0x001ec1, 0x001ec0, 0x001ec0},
+ {0x001ec1, 0x001ec1, 0x001ec0, 0x001ec0},
+ {0x001ec2, 0x001ec3, 0x001ec2, 0x001ec2},
+ {0x001ec3, 0x001ec3, 0x001ec2, 0x001ec2},
+ {0x001ec4, 0x001ec5, 0x001ec4, 0x001ec4},
+ {0x001ec5, 0x001ec5, 0x001ec4, 0x001ec4},
+ {0x001ec6, 0x001ec7, 0x001ec6, 0x001ec6},
+ {0x001ec7, 0x001ec7, 0x001ec6, 0x001ec6},
+ {0x001ec8, 0x001ec9, 0x001ec8, 0x001ec8},
+ {0x001ec9, 0x001ec9, 0x001ec8, 0x001ec8},
+ {0x001eca, 0x001ecb, 0x001eca, 0x001eca},
+ {0x001ecb, 0x001ecb, 0x001eca, 0x001eca},
+ {0x001ecc, 0x001ecd, 0x001ecc, 0x001ecc},
+ {0x001ecd, 0x001ecd, 0x001ecc, 0x001ecc},
+ {0x001ece, 0x001ecf, 0x001ece, 0x001ece},
+ {0x001ecf, 0x001ecf, 0x001ece, 0x001ece},
+ {0x001ed0, 0x001ed1, 0x001ed0, 0x001ed0},
+ {0x001ed1, 0x001ed1, 0x001ed0, 0x001ed0},
+ {0x001ed2, 0x001ed3, 0x001ed2, 0x001ed2},
+ {0x001ed3, 0x001ed3, 0x001ed2, 0x001ed2},
+ {0x001ed4, 0x001ed5, 0x001ed4, 0x001ed4},
+ {0x001ed5, 0x001ed5, 0x001ed4, 0x001ed4},
+ {0x001ed6, 0x001ed7, 0x001ed6, 0x001ed6},
+ {0x001ed7, 0x001ed7, 0x001ed6, 0x001ed6},
+ {0x001ed8, 0x001ed9, 0x001ed8, 0x001ed8},
+ {0x001ed9, 0x001ed9, 0x001ed8, 0x001ed8},
+ {0x001eda, 0x001edb, 0x001eda, 0x001eda},
+ {0x001edb, 0x001edb, 0x001eda, 0x001eda},
+ {0x001edc, 0x001edd, 0x001edc, 0x001edc},
+ {0x001edd, 0x001edd, 0x001edc, 0x001edc},
+ {0x001ede, 0x001edf, 0x001ede, 0x001ede},
+ {0x001edf, 0x001edf, 0x001ede, 0x001ede},
+ {0x001ee0, 0x001ee1, 0x001ee0, 0x001ee0},
+ {0x001ee1, 0x001ee1, 0x001ee0, 0x001ee0},
+ {0x001ee2, 0x001ee3, 0x001ee2, 0x001ee2},
+ {0x001ee3, 0x001ee3, 0x001ee2, 0x001ee2},
+ {0x001ee4, 0x001ee5, 0x001ee4, 0x001ee4},
+ {0x001ee5, 0x001ee5, 0x001ee4, 0x001ee4},
+ {0x001ee6, 0x001ee7, 0x001ee6, 0x001ee6},
+ {0x001ee7, 0x001ee7, 0x001ee6, 0x001ee6},
+ {0x001ee8, 0x001ee9, 0x001ee8, 0x001ee8},
+ {0x001ee9, 0x001ee9, 0x001ee8, 0x001ee8},
+ {0x001eea, 0x001eeb, 0x001eea, 0x001eea},
+ {0x001eeb, 0x001eeb, 0x001eea, 0x001eea},
+ {0x001eec, 0x001eed, 0x001eec, 0x001eec},
+ {0x001eed, 0x001eed, 0x001eec, 0x001eec},
+ {0x001eee, 0x001eef, 0x001eee, 0x001eee},
+ {0x001eef, 0x001eef, 0x001eee, 0x001eee},
+ {0x001ef0, 0x001ef1, 0x001ef0, 0x001ef0},
+ {0x001ef1, 0x001ef1, 0x001ef0, 0x001ef0},
+ {0x001ef2, 0x001ef3, 0x001ef2, 0x001ef2},
+ {0x001ef3, 0x001ef3, 0x001ef2, 0x001ef2},
+ {0x001ef4, 0x001ef5, 0x001ef4, 0x001ef4},
+ {0x001ef5, 0x001ef5, 0x001ef4, 0x001ef4},
+ {0x001ef6, 0x001ef7, 0x001ef6, 0x001ef6},
+ {0x001ef7, 0x001ef7, 0x001ef6, 0x001ef6},
+ {0x001ef8, 0x001ef9, 0x001ef8, 0x001ef8},
+ {0x001ef9, 0x001ef9, 0x001ef8, 0x001ef8},
+ {0x001efa, 0x001efb, 0x001efa, 0x001efa},
+ {0x001efb, 0x001efb, 0x001efa, 0x001efa},
+ {0x001efc, 0x001efd, 0x001efc, 0x001efc},
+ {0x001efd, 0x001efd, 0x001efc, 0x001efc},
+ {0x001efe, 0x001eff, 0x001efe, 0x001efe},
+ {0x001eff, 0x001eff, 0x001efe, 0x001efe},
+ {0x001f00, 0x001f00, 0x001f08, 0x001f08},
+ {0x001f01, 0x001f01, 0x001f09, 0x001f09},
+ {0x001f02, 0x001f02, 0x001f0a, 0x001f0a},
+ {0x001f03, 0x001f03, 0x001f0b, 0x001f0b},
+ {0x001f04, 0x001f04, 0x001f0c, 0x001f0c},
+ {0x001f05, 0x001f05, 0x001f0d, 0x001f0d},
+ {0x001f06, 0x001f06, 0x001f0e, 0x001f0e},
+ {0x001f07, 0x001f07, 0x001f0f, 0x001f0f},
+ {0x001f08, 0x001f00, 0x001f08, 0x001f08},
+ {0x001f09, 0x001f01, 0x001f09, 0x001f09},
+ {0x001f0a, 0x001f02, 0x001f0a, 0x001f0a},
+ {0x001f0b, 0x001f03, 0x001f0b, 0x001f0b},
+ {0x001f0c, 0x001f04, 0x001f0c, 0x001f0c},
+ {0x001f0d, 0x001f05, 0x001f0d, 0x001f0d},
+ {0x001f0e, 0x001f06, 0x001f0e, 0x001f0e},
+ {0x001f0f, 0x001f07, 0x001f0f, 0x001f0f},
+ {0x001f10, 0x001f10, 0x001f18, 0x001f18},
+ {0x001f11, 0x001f11, 0x001f19, 0x001f19},
+ {0x001f12, 0x001f12, 0x001f1a, 0x001f1a},
+ {0x001f13, 0x001f13, 0x001f1b, 0x001f1b},
+ {0x001f14, 0x001f14, 0x001f1c, 0x001f1c},
+ {0x001f15, 0x001f15, 0x001f1d, 0x001f1d},
+ {0x001f18, 0x001f10, 0x001f18, 0x001f18},
+ {0x001f19, 0x001f11, 0x001f19, 0x001f19},
+ {0x001f1a, 0x001f12, 0x001f1a, 0x001f1a},
+ {0x001f1b, 0x001f13, 0x001f1b, 0x001f1b},
+ {0x001f1c, 0x001f14, 0x001f1c, 0x001f1c},
+ {0x001f1d, 0x001f15, 0x001f1d, 0x001f1d},
+ {0x001f20, 0x001f20, 0x001f28, 0x001f28},
+ {0x001f21, 0x001f21, 0x001f29, 0x001f29},
+ {0x001f22, 0x001f22, 0x001f2a, 0x001f2a},
+ {0x001f23, 0x001f23, 0x001f2b, 0x001f2b},
+ {0x001f24, 0x001f24, 0x001f2c, 0x001f2c},
+ {0x001f25, 0x001f25, 0x001f2d, 0x001f2d},
+ {0x001f26, 0x001f26, 0x001f2e, 0x001f2e},
+ {0x001f27, 0x001f27, 0x001f2f, 0x001f2f},
+ {0x001f28, 0x001f20, 0x001f28, 0x001f28},
+ {0x001f29, 0x001f21, 0x001f29, 0x001f29},
+ {0x001f2a, 0x001f22, 0x001f2a, 0x001f2a},
+ {0x001f2b, 0x001f23, 0x001f2b, 0x001f2b},
+ {0x001f2c, 0x001f24, 0x001f2c, 0x001f2c},
+ {0x001f2d, 0x001f25, 0x001f2d, 0x001f2d},
+ {0x001f2e, 0x001f26, 0x001f2e, 0x001f2e},
+ {0x001f2f, 0x001f27, 0x001f2f, 0x001f2f},
+ {0x001f30, 0x001f30, 0x001f38, 0x001f38},
+ {0x001f31, 0x001f31, 0x001f39, 0x001f39},
+ {0x001f32, 0x001f32, 0x001f3a, 0x001f3a},
+ {0x001f33, 0x001f33, 0x001f3b, 0x001f3b},
+ {0x001f34, 0x001f34, 0x001f3c, 0x001f3c},
+ {0x001f35, 0x001f35, 0x001f3d, 0x001f3d},
+ {0x001f36, 0x001f36, 0x001f3e, 0x001f3e},
+ {0x001f37, 0x001f37, 0x001f3f, 0x001f3f},
+ {0x001f38, 0x001f30, 0x001f38, 0x001f38},
+ {0x001f39, 0x001f31, 0x001f39, 0x001f39},
+ {0x001f3a, 0x001f32, 0x001f3a, 0x001f3a},
+ {0x001f3b, 0x001f33, 0x001f3b, 0x001f3b},
+ {0x001f3c, 0x001f34, 0x001f3c, 0x001f3c},
+ {0x001f3d, 0x001f35, 0x001f3d, 0x001f3d},
+ {0x001f3e, 0x001f36, 0x001f3e, 0x001f3e},
+ {0x001f3f, 0x001f37, 0x001f3f, 0x001f3f},
+ {0x001f40, 0x001f40, 0x001f48, 0x001f48},
+ {0x001f41, 0x001f41, 0x001f49, 0x001f49},
+ {0x001f42, 0x001f42, 0x001f4a, 0x001f4a},
+ {0x001f43, 0x001f43, 0x001f4b, 0x001f4b},
+ {0x001f44, 0x001f44, 0x001f4c, 0x001f4c},
+ {0x001f45, 0x001f45, 0x001f4d, 0x001f4d},
+ {0x001f48, 0x001f40, 0x001f48, 0x001f48},
+ {0x001f49, 0x001f41, 0x001f49, 0x001f49},
+ {0x001f4a, 0x001f42, 0x001f4a, 0x001f4a},
+ {0x001f4b, 0x001f43, 0x001f4b, 0x001f4b},
+ {0x001f4c, 0x001f44, 0x001f4c, 0x001f4c},
+ {0x001f4d, 0x001f45, 0x001f4d, 0x001f4d},
+ {0x001f51, 0x001f51, 0x001f59, 0x001f59},
+ {0x001f53, 0x001f53, 0x001f5b, 0x001f5b},
+ {0x001f55, 0x001f55, 0x001f5d, 0x001f5d},
+ {0x001f57, 0x001f57, 0x001f5f, 0x001f5f},
+ {0x001f59, 0x001f51, 0x001f59, 0x001f59},
+ {0x001f5b, 0x001f53, 0x001f5b, 0x001f5b},
+ {0x001f5d, 0x001f55, 0x001f5d, 0x001f5d},
+ {0x001f5f, 0x001f57, 0x001f5f, 0x001f5f},
+ {0x001f60, 0x001f60, 0x001f68, 0x001f68},
+ {0x001f61, 0x001f61, 0x001f69, 0x001f69},
+ {0x001f62, 0x001f62, 0x001f6a, 0x001f6a},
+ {0x001f63, 0x001f63, 0x001f6b, 0x001f6b},
+ {0x001f64, 0x001f64, 0x001f6c, 0x001f6c},
+ {0x001f65, 0x001f65, 0x001f6d, 0x001f6d},
+ {0x001f66, 0x001f66, 0x001f6e, 0x001f6e},
+ {0x001f67, 0x001f67, 0x001f6f, 0x001f6f},
+ {0x001f68, 0x001f60, 0x001f68, 0x001f68},
+ {0x001f69, 0x001f61, 0x001f69, 0x001f69},
+ {0x001f6a, 0x001f62, 0x001f6a, 0x001f6a},
+ {0x001f6b, 0x001f63, 0x001f6b, 0x001f6b},
+ {0x001f6c, 0x001f64, 0x001f6c, 0x001f6c},
+ {0x001f6d, 0x001f65, 0x001f6d, 0x001f6d},
+ {0x001f6e, 0x001f66, 0x001f6e, 0x001f6e},
+ {0x001f6f, 0x001f67, 0x001f6f, 0x001f6f},
+ {0x001f70, 0x001f70, 0x001fba, 0x001fba},
+ {0x001f71, 0x001f71, 0x001fbb, 0x001fbb},
+ {0x001f72, 0x001f72, 0x001fc8, 0x001fc8},
+ {0x001f73, 0x001f73, 0x001fc9, 0x001fc9},
+ {0x001f74, 0x001f74, 0x001fca, 0x001fca},
+ {0x001f75, 0x001f75, 0x001fcb, 0x001fcb},
+ {0x001f76, 0x001f76, 0x001fda, 0x001fda},
+ {0x001f77, 0x001f77, 0x001fdb, 0x001fdb},
+ {0x001f78, 0x001f78, 0x001ff8, 0x001ff8},
+ {0x001f79, 0x001f79, 0x001ff9, 0x001ff9},
+ {0x001f7a, 0x001f7a, 0x001fea, 0x001fea},
+ {0x001f7b, 0x001f7b, 0x001feb, 0x001feb},
+ {0x001f7c, 0x001f7c, 0x001ffa, 0x001ffa},
+ {0x001f7d, 0x001f7d, 0x001ffb, 0x001ffb},
+ {0x001f80, 0x001f80, 0x001f88, 0x001f88},
+ {0x001f81, 0x001f81, 0x001f89, 0x001f89},
+ {0x001f82, 0x001f82, 0x001f8a, 0x001f8a},
+ {0x001f83, 0x001f83, 0x001f8b, 0x001f8b},
+ {0x001f84, 0x001f84, 0x001f8c, 0x001f8c},
+ {0x001f85, 0x001f85, 0x001f8d, 0x001f8d},
+ {0x001f86, 0x001f86, 0x001f8e, 0x001f8e},
+ {0x001f87, 0x001f87, 0x001f8f, 0x001f8f},
+ {0x001f88, 0x001f80, 0x001f88, 0x001f88},
+ {0x001f89, 0x001f81, 0x001f89, 0x001f89},
+ {0x001f8a, 0x001f82, 0x001f8a, 0x001f8a},
+ {0x001f8b, 0x001f83, 0x001f8b, 0x001f8b},
+ {0x001f8c, 0x001f84, 0x001f8c, 0x001f8c},
+ {0x001f8d, 0x001f85, 0x001f8d, 0x001f8d},
+ {0x001f8e, 0x001f86, 0x001f8e, 0x001f8e},
+ {0x001f8f, 0x001f87, 0x001f8f, 0x001f8f},
+ {0x001f90, 0x001f90, 0x001f98, 0x001f98},
+ {0x001f91, 0x001f91, 0x001f99, 0x001f99},
+ {0x001f92, 0x001f92, 0x001f9a, 0x001f9a},
+ {0x001f93, 0x001f93, 0x001f9b, 0x001f9b},
+ {0x001f94, 0x001f94, 0x001f9c, 0x001f9c},
+ {0x001f95, 0x001f95, 0x001f9d, 0x001f9d},
+ {0x001f96, 0x001f96, 0x001f9e, 0x001f9e},
+ {0x001f97, 0x001f97, 0x001f9f, 0x001f9f},
+ {0x001f98, 0x001f90, 0x001f98, 0x001f98},
+ {0x001f99, 0x001f91, 0x001f99, 0x001f99},
+ {0x001f9a, 0x001f92, 0x001f9a, 0x001f9a},
+ {0x001f9b, 0x001f93, 0x001f9b, 0x001f9b},
+ {0x001f9c, 0x001f94, 0x001f9c, 0x001f9c},
+ {0x001f9d, 0x001f95, 0x001f9d, 0x001f9d},
+ {0x001f9e, 0x001f96, 0x001f9e, 0x001f9e},
+ {0x001f9f, 0x001f97, 0x001f9f, 0x001f9f},
+ {0x001fa0, 0x001fa0, 0x001fa8, 0x001fa8},
+ {0x001fa1, 0x001fa1, 0x001fa9, 0x001fa9},
+ {0x001fa2, 0x001fa2, 0x001faa, 0x001faa},
+ {0x001fa3, 0x001fa3, 0x001fab, 0x001fab},
+ {0x001fa4, 0x001fa4, 0x001fac, 0x001fac},
+ {0x001fa5, 0x001fa5, 0x001fad, 0x001fad},
+ {0x001fa6, 0x001fa6, 0x001fae, 0x001fae},
+ {0x001fa7, 0x001fa7, 0x001faf, 0x001faf},
+ {0x001fa8, 0x001fa0, 0x001fa8, 0x001fa8},
+ {0x001fa9, 0x001fa1, 0x001fa9, 0x001fa9},
+ {0x001faa, 0x001fa2, 0x001faa, 0x001faa},
+ {0x001fab, 0x001fa3, 0x001fab, 0x001fab},
+ {0x001fac, 0x001fa4, 0x001fac, 0x001fac},
+ {0x001fad, 0x001fa5, 0x001fad, 0x001fad},
+ {0x001fae, 0x001fa6, 0x001fae, 0x001fae},
+ {0x001faf, 0x001fa7, 0x001faf, 0x001faf},
+ {0x001fb0, 0x001fb0, 0x001fb8, 0x001fb8},
+ {0x001fb1, 0x001fb1, 0x001fb9, 0x001fb9},
+ {0x001fb3, 0x001fb3, 0x001fbc, 0x001fbc},
+ {0x001fb8, 0x001fb0, 0x001fb8, 0x001fb8},
+ {0x001fb9, 0x001fb1, 0x001fb9, 0x001fb9},
+ {0x001fba, 0x001f70, 0x001fba, 0x001fba},
+ {0x001fbb, 0x001f71, 0x001fbb, 0x001fbb},
+ {0x001fbc, 0x001fb3, 0x001fbc, 0x001fbc},
+ {0x001fbe, 0x001fbe, 0x000399, 0x000399},
+ {0x001fc3, 0x001fc3, 0x001fcc, 0x001fcc},
+ {0x001fc8, 0x001f72, 0x001fc8, 0x001fc8},
+ {0x001fc9, 0x001f73, 0x001fc9, 0x001fc9},
+ {0x001fca, 0x001f74, 0x001fca, 0x001fca},
+ {0x001fcb, 0x001f75, 0x001fcb, 0x001fcb},
+ {0x001fcc, 0x001fc3, 0x001fcc, 0x001fcc},
+ {0x001fd0, 0x001fd0, 0x001fd8, 0x001fd8},
+ {0x001fd1, 0x001fd1, 0x001fd9, 0x001fd9},
+ {0x001fd8, 0x001fd0, 0x001fd8, 0x001fd8},
+ {0x001fd9, 0x001fd1, 0x001fd9, 0x001fd9},
+ {0x001fda, 0x001f76, 0x001fda, 0x001fda},
+ {0x001fdb, 0x001f77, 0x001fdb, 0x001fdb},
+ {0x001fe0, 0x001fe0, 0x001fe8, 0x001fe8},
+ {0x001fe1, 0x001fe1, 0x001fe9, 0x001fe9},
+ {0x001fe5, 0x001fe5, 0x001fec, 0x001fec},
+ {0x001fe8, 0x001fe0, 0x001fe8, 0x001fe8},
+ {0x001fe9, 0x001fe1, 0x001fe9, 0x001fe9},
+ {0x001fea, 0x001f7a, 0x001fea, 0x001fea},
+ {0x001feb, 0x001f7b, 0x001feb, 0x001feb},
+ {0x001fec, 0x001fe5, 0x001fec, 0x001fec},
+ {0x001ff3, 0x001ff3, 0x001ffc, 0x001ffc},
+ {0x001ff8, 0x001f78, 0x001ff8, 0x001ff8},
+ {0x001ff9, 0x001f79, 0x001ff9, 0x001ff9},
+ {0x001ffa, 0x001f7c, 0x001ffa, 0x001ffa},
+ {0x001ffb, 0x001f7d, 0x001ffb, 0x001ffb},
+ {0x001ffc, 0x001ff3, 0x001ffc, 0x001ffc},
+ {0x002126, 0x0003c9, 0x002126, 0x002126},
+ {0x00212a, 0x00006b, 0x00212a, 0x00212a},
+ {0x00212b, 0x0000e5, 0x00212b, 0x00212b},
+ {0x002132, 0x00214e, 0x002132, 0x002132},
+ {0x00214e, 0x00214e, 0x002132, 0x002132},
+ {0x002160, 0x002170, 0x002160, 0x002160},
+ {0x002161, 0x002171, 0x002161, 0x002161},
+ {0x002162, 0x002172, 0x002162, 0x002162},
+ {0x002163, 0x002173, 0x002163, 0x002163},
+ {0x002164, 0x002174, 0x002164, 0x002164},
+ {0x002165, 0x002175, 0x002165, 0x002165},
+ {0x002166, 0x002176, 0x002166, 0x002166},
+ {0x002167, 0x002177, 0x002167, 0x002167},
+ {0x002168, 0x002178, 0x002168, 0x002168},
+ {0x002169, 0x002179, 0x002169, 0x002169},
+ {0x00216a, 0x00217a, 0x00216a, 0x00216a},
+ {0x00216b, 0x00217b, 0x00216b, 0x00216b},
+ {0x00216c, 0x00217c, 0x00216c, 0x00216c},
+ {0x00216d, 0x00217d, 0x00216d, 0x00216d},
+ {0x00216e, 0x00217e, 0x00216e, 0x00216e},
+ {0x00216f, 0x00217f, 0x00216f, 0x00216f},
+ {0x002170, 0x002170, 0x002160, 0x002160},
+ {0x002171, 0x002171, 0x002161, 0x002161},
+ {0x002172, 0x002172, 0x002162, 0x002162},
+ {0x002173, 0x002173, 0x002163, 0x002163},
+ {0x002174, 0x002174, 0x002164, 0x002164},
+ {0x002175, 0x002175, 0x002165, 0x002165},
+ {0x002176, 0x002176, 0x002166, 0x002166},
+ {0x002177, 0x002177, 0x002167, 0x002167},
+ {0x002178, 0x002178, 0x002168, 0x002168},
+ {0x002179, 0x002179, 0x002169, 0x002169},
+ {0x00217a, 0x00217a, 0x00216a, 0x00216a},
+ {0x00217b, 0x00217b, 0x00216b, 0x00216b},
+ {0x00217c, 0x00217c, 0x00216c, 0x00216c},
+ {0x00217d, 0x00217d, 0x00216d, 0x00216d},
+ {0x00217e, 0x00217e, 0x00216e, 0x00216e},
+ {0x00217f, 0x00217f, 0x00216f, 0x00216f},
+ {0x002183, 0x002184, 0x002183, 0x002183},
+ {0x002184, 0x002184, 0x002183, 0x002183},
+ {0x0024b6, 0x0024d0, 0x0024b6, 0x0024b6},
+ {0x0024b7, 0x0024d1, 0x0024b7, 0x0024b7},
+ {0x0024b8, 0x0024d2, 0x0024b8, 0x0024b8},
+ {0x0024b9, 0x0024d3, 0x0024b9, 0x0024b9},
+ {0x0024ba, 0x0024d4, 0x0024ba, 0x0024ba},
+ {0x0024bb, 0x0024d5, 0x0024bb, 0x0024bb},
+ {0x0024bc, 0x0024d6, 0x0024bc, 0x0024bc},
+ {0x0024bd, 0x0024d7, 0x0024bd, 0x0024bd},
+ {0x0024be, 0x0024d8, 0x0024be, 0x0024be},
+ {0x0024bf, 0x0024d9, 0x0024bf, 0x0024bf},
+ {0x0024c0, 0x0024da, 0x0024c0, 0x0024c0},
+ {0x0024c1, 0x0024db, 0x0024c1, 0x0024c1},
+ {0x0024c2, 0x0024dc, 0x0024c2, 0x0024c2},
+ {0x0024c3, 0x0024dd, 0x0024c3, 0x0024c3},
+ {0x0024c4, 0x0024de, 0x0024c4, 0x0024c4},
+ {0x0024c5, 0x0024df, 0x0024c5, 0x0024c5},
+ {0x0024c6, 0x0024e0, 0x0024c6, 0x0024c6},
+ {0x0024c7, 0x0024e1, 0x0024c7, 0x0024c7},
+ {0x0024c8, 0x0024e2, 0x0024c8, 0x0024c8},
+ {0x0024c9, 0x0024e3, 0x0024c9, 0x0024c9},
+ {0x0024ca, 0x0024e4, 0x0024ca, 0x0024ca},
+ {0x0024cb, 0x0024e5, 0x0024cb, 0x0024cb},
+ {0x0024cc, 0x0024e6, 0x0024cc, 0x0024cc},
+ {0x0024cd, 0x0024e7, 0x0024cd, 0x0024cd},
+ {0x0024ce, 0x0024e8, 0x0024ce, 0x0024ce},
+ {0x0024cf, 0x0024e9, 0x0024cf, 0x0024cf},
+ {0x0024d0, 0x0024d0, 0x0024b6, 0x0024b6},
+ {0x0024d1, 0x0024d1, 0x0024b7, 0x0024b7},
+ {0x0024d2, 0x0024d2, 0x0024b8, 0x0024b8},
+ {0x0024d3, 0x0024d3, 0x0024b9, 0x0024b9},
+ {0x0024d4, 0x0024d4, 0x0024ba, 0x0024ba},
+ {0x0024d5, 0x0024d5, 0x0024bb, 0x0024bb},
+ {0x0024d6, 0x0024d6, 0x0024bc, 0x0024bc},
+ {0x0024d7, 0x0024d7, 0x0024bd, 0x0024bd},
+ {0x0024d8, 0x0024d8, 0x0024be, 0x0024be},
+ {0x0024d9, 0x0024d9, 0x0024bf, 0x0024bf},
+ {0x0024da, 0x0024da, 0x0024c0, 0x0024c0},
+ {0x0024db, 0x0024db, 0x0024c1, 0x0024c1},
+ {0x0024dc, 0x0024dc, 0x0024c2, 0x0024c2},
+ {0x0024dd, 0x0024dd, 0x0024c3, 0x0024c3},
+ {0x0024de, 0x0024de, 0x0024c4, 0x0024c4},
+ {0x0024df, 0x0024df, 0x0024c5, 0x0024c5},
+ {0x0024e0, 0x0024e0, 0x0024c6, 0x0024c6},
+ {0x0024e1, 0x0024e1, 0x0024c7, 0x0024c7},
+ {0x0024e2, 0x0024e2, 0x0024c8, 0x0024c8},
+ {0x0024e3, 0x0024e3, 0x0024c9, 0x0024c9},
+ {0x0024e4, 0x0024e4, 0x0024ca, 0x0024ca},
+ {0x0024e5, 0x0024e5, 0x0024cb, 0x0024cb},
+ {0x0024e6, 0x0024e6, 0x0024cc, 0x0024cc},
+ {0x0024e7, 0x0024e7, 0x0024cd, 0x0024cd},
+ {0x0024e8, 0x0024e8, 0x0024ce, 0x0024ce},
+ {0x0024e9, 0x0024e9, 0x0024cf, 0x0024cf},
+ {0x002c00, 0x002c30, 0x002c00, 0x002c00},
+ {0x002c01, 0x002c31, 0x002c01, 0x002c01},
+ {0x002c02, 0x002c32, 0x002c02, 0x002c02},
+ {0x002c03, 0x002c33, 0x002c03, 0x002c03},
+ {0x002c04, 0x002c34, 0x002c04, 0x002c04},
+ {0x002c05, 0x002c35, 0x002c05, 0x002c05},
+ {0x002c06, 0x002c36, 0x002c06, 0x002c06},
+ {0x002c07, 0x002c37, 0x002c07, 0x002c07},
+ {0x002c08, 0x002c38, 0x002c08, 0x002c08},
+ {0x002c09, 0x002c39, 0x002c09, 0x002c09},
+ {0x002c0a, 0x002c3a, 0x002c0a, 0x002c0a},
+ {0x002c0b, 0x002c3b, 0x002c0b, 0x002c0b},
+ {0x002c0c, 0x002c3c, 0x002c0c, 0x002c0c},
+ {0x002c0d, 0x002c3d, 0x002c0d, 0x002c0d},
+ {0x002c0e, 0x002c3e, 0x002c0e, 0x002c0e},
+ {0x002c0f, 0x002c3f, 0x002c0f, 0x002c0f},
+ {0x002c10, 0x002c40, 0x002c10, 0x002c10},
+ {0x002c11, 0x002c41, 0x002c11, 0x002c11},
+ {0x002c12, 0x002c42, 0x002c12, 0x002c12},
+ {0x002c13, 0x002c43, 0x002c13, 0x002c13},
+ {0x002c14, 0x002c44, 0x002c14, 0x002c14},
+ {0x002c15, 0x002c45, 0x002c15, 0x002c15},
+ {0x002c16, 0x002c46, 0x002c16, 0x002c16},
+ {0x002c17, 0x002c47, 0x002c17, 0x002c17},
+ {0x002c18, 0x002c48, 0x002c18, 0x002c18},
+ {0x002c19, 0x002c49, 0x002c19, 0x002c19},
+ {0x002c1a, 0x002c4a, 0x002c1a, 0x002c1a},
+ {0x002c1b, 0x002c4b, 0x002c1b, 0x002c1b},
+ {0x002c1c, 0x002c4c, 0x002c1c, 0x002c1c},
+ {0x002c1d, 0x002c4d, 0x002c1d, 0x002c1d},
+ {0x002c1e, 0x002c4e, 0x002c1e, 0x002c1e},
+ {0x002c1f, 0x002c4f, 0x002c1f, 0x002c1f},
+ {0x002c20, 0x002c50, 0x002c20, 0x002c20},
+ {0x002c21, 0x002c51, 0x002c21, 0x002c21},
+ {0x002c22, 0x002c52, 0x002c22, 0x002c22},
+ {0x002c23, 0x002c53, 0x002c23, 0x002c23},
+ {0x002c24, 0x002c54, 0x002c24, 0x002c24},
+ {0x002c25, 0x002c55, 0x002c25, 0x002c25},
+ {0x002c26, 0x002c56, 0x002c26, 0x002c26},
+ {0x002c27, 0x002c57, 0x002c27, 0x002c27},
+ {0x002c28, 0x002c58, 0x002c28, 0x002c28},
+ {0x002c29, 0x002c59, 0x002c29, 0x002c29},
+ {0x002c2a, 0x002c5a, 0x002c2a, 0x002c2a},
+ {0x002c2b, 0x002c5b, 0x002c2b, 0x002c2b},
+ {0x002c2c, 0x002c5c, 0x002c2c, 0x002c2c},
+ {0x002c2d, 0x002c5d, 0x002c2d, 0x002c2d},
+ {0x002c2e, 0x002c5e, 0x002c2e, 0x002c2e},
+ {0x002c2f, 0x002c5f, 0x002c2f, 0x002c2f},
+ {0x002c30, 0x002c30, 0x002c00, 0x002c00},
+ {0x002c31, 0x002c31, 0x002c01, 0x002c01},
+ {0x002c32, 0x002c32, 0x002c02, 0x002c02},
+ {0x002c33, 0x002c33, 0x002c03, 0x002c03},
+ {0x002c34, 0x002c34, 0x002c04, 0x002c04},
+ {0x002c35, 0x002c35, 0x002c05, 0x002c05},
+ {0x002c36, 0x002c36, 0x002c06, 0x002c06},
+ {0x002c37, 0x002c37, 0x002c07, 0x002c07},
+ {0x002c38, 0x002c38, 0x002c08, 0x002c08},
+ {0x002c39, 0x002c39, 0x002c09, 0x002c09},
+ {0x002c3a, 0x002c3a, 0x002c0a, 0x002c0a},
+ {0x002c3b, 0x002c3b, 0x002c0b, 0x002c0b},
+ {0x002c3c, 0x002c3c, 0x002c0c, 0x002c0c},
+ {0x002c3d, 0x002c3d, 0x002c0d, 0x002c0d},
+ {0x002c3e, 0x002c3e, 0x002c0e, 0x002c0e},
+ {0x002c3f, 0x002c3f, 0x002c0f, 0x002c0f},
+ {0x002c40, 0x002c40, 0x002c10, 0x002c10},
+ {0x002c41, 0x002c41, 0x002c11, 0x002c11},
+ {0x002c42, 0x002c42, 0x002c12, 0x002c12},
+ {0x002c43, 0x002c43, 0x002c13, 0x002c13},
+ {0x002c44, 0x002c44, 0x002c14, 0x002c14},
+ {0x002c45, 0x002c45, 0x002c15, 0x002c15},
+ {0x002c46, 0x002c46, 0x002c16, 0x002c16},
+ {0x002c47, 0x002c47, 0x002c17, 0x002c17},
+ {0x002c48, 0x002c48, 0x002c18, 0x002c18},
+ {0x002c49, 0x002c49, 0x002c19, 0x002c19},
+ {0x002c4a, 0x002c4a, 0x002c1a, 0x002c1a},
+ {0x002c4b, 0x002c4b, 0x002c1b, 0x002c1b},
+ {0x002c4c, 0x002c4c, 0x002c1c, 0x002c1c},
+ {0x002c4d, 0x002c4d, 0x002c1d, 0x002c1d},
+ {0x002c4e, 0x002c4e, 0x002c1e, 0x002c1e},
+ {0x002c4f, 0x002c4f, 0x002c1f, 0x002c1f},
+ {0x002c50, 0x002c50, 0x002c20, 0x002c20},
+ {0x002c51, 0x002c51, 0x002c21, 0x002c21},
+ {0x002c52, 0x002c52, 0x002c22, 0x002c22},
+ {0x002c53, 0x002c53, 0x002c23, 0x002c23},
+ {0x002c54, 0x002c54, 0x002c24, 0x002c24},
+ {0x002c55, 0x002c55, 0x002c25, 0x002c25},
+ {0x002c56, 0x002c56, 0x002c26, 0x002c26},
+ {0x002c57, 0x002c57, 0x002c27, 0x002c27},
+ {0x002c58, 0x002c58, 0x002c28, 0x002c28},
+ {0x002c59, 0x002c59, 0x002c29, 0x002c29},
+ {0x002c5a, 0x002c5a, 0x002c2a, 0x002c2a},
+ {0x002c5b, 0x002c5b, 0x002c2b, 0x002c2b},
+ {0x002c5c, 0x002c5c, 0x002c2c, 0x002c2c},
+ {0x002c5d, 0x002c5d, 0x002c2d, 0x002c2d},
+ {0x002c5e, 0x002c5e, 0x002c2e, 0x002c2e},
+ {0x002c5f, 0x002c5f, 0x002c2f, 0x002c2f},
+ {0x002c60, 0x002c61, 0x002c60, 0x002c60},
+ {0x002c61, 0x002c61, 0x002c60, 0x002c60},
+ {0x002c62, 0x00026b, 0x002c62, 0x002c62},
+ {0x002c63, 0x001d7d, 0x002c63, 0x002c63},
+ {0x002c64, 0x00027d, 0x002c64, 0x002c64},
+ {0x002c65, 0x002c65, 0x00023a, 0x00023a},
+ {0x002c66, 0x002c66, 0x00023e, 0x00023e},
+ {0x002c67, 0x002c68, 0x002c67, 0x002c67},
+ {0x002c68, 0x002c68, 0x002c67, 0x002c67},
+ {0x002c69, 0x002c6a, 0x002c69, 0x002c69},
+ {0x002c6a, 0x002c6a, 0x002c69, 0x002c69},
+ {0x002c6b, 0x002c6c, 0x002c6b, 0x002c6b},
+ {0x002c6c, 0x002c6c, 0x002c6b, 0x002c6b},
+ {0x002c6d, 0x000251, 0x002c6d, 0x002c6d},
+ {0x002c6e, 0x000271, 0x002c6e, 0x002c6e},
+ {0x002c6f, 0x000250, 0x002c6f, 0x002c6f},
+ {0x002c70, 0x000252, 0x002c70, 0x002c70},
+ {0x002c72, 0x002c73, 0x002c72, 0x002c72},
+ {0x002c73, 0x002c73, 0x002c72, 0x002c72},
+ {0x002c75, 0x002c76, 0x002c75, 0x002c75},
+ {0x002c76, 0x002c76, 0x002c75, 0x002c75},
+ {0x002c7e, 0x00023f, 0x002c7e, 0x002c7e},
+ {0x002c7f, 0x000240, 0x002c7f, 0x002c7f},
+ {0x002c80, 0x002c81, 0x002c80, 0x002c80},
+ {0x002c81, 0x002c81, 0x002c80, 0x002c80},
+ {0x002c82, 0x002c83, 0x002c82, 0x002c82},
+ {0x002c83, 0x002c83, 0x002c82, 0x002c82},
+ {0x002c84, 0x002c85, 0x002c84, 0x002c84},
+ {0x002c85, 0x002c85, 0x002c84, 0x002c84},
+ {0x002c86, 0x002c87, 0x002c86, 0x002c86},
+ {0x002c87, 0x002c87, 0x002c86, 0x002c86},
+ {0x002c88, 0x002c89, 0x002c88, 0x002c88},
+ {0x002c89, 0x002c89, 0x002c88, 0x002c88},
+ {0x002c8a, 0x002c8b, 0x002c8a, 0x002c8a},
+ {0x002c8b, 0x002c8b, 0x002c8a, 0x002c8a},
+ {0x002c8c, 0x002c8d, 0x002c8c, 0x002c8c},
+ {0x002c8d, 0x002c8d, 0x002c8c, 0x002c8c},
+ {0x002c8e, 0x002c8f, 0x002c8e, 0x002c8e},
+ {0x002c8f, 0x002c8f, 0x002c8e, 0x002c8e},
+ {0x002c90, 0x002c91, 0x002c90, 0x002c90},
+ {0x002c91, 0x002c91, 0x002c90, 0x002c90},
+ {0x002c92, 0x002c93, 0x002c92, 0x002c92},
+ {0x002c93, 0x002c93, 0x002c92, 0x002c92},
+ {0x002c94, 0x002c95, 0x002c94, 0x002c94},
+ {0x002c95, 0x002c95, 0x002c94, 0x002c94},
+ {0x002c96, 0x002c97, 0x002c96, 0x002c96},
+ {0x002c97, 0x002c97, 0x002c96, 0x002c96},
+ {0x002c98, 0x002c99, 0x002c98, 0x002c98},
+ {0x002c99, 0x002c99, 0x002c98, 0x002c98},
+ {0x002c9a, 0x002c9b, 0x002c9a, 0x002c9a},
+ {0x002c9b, 0x002c9b, 0x002c9a, 0x002c9a},
+ {0x002c9c, 0x002c9d, 0x002c9c, 0x002c9c},
+ {0x002c9d, 0x002c9d, 0x002c9c, 0x002c9c},
+ {0x002c9e, 0x002c9f, 0x002c9e, 0x002c9e},
+ {0x002c9f, 0x002c9f, 0x002c9e, 0x002c9e},
+ {0x002ca0, 0x002ca1, 0x002ca0, 0x002ca0},
+ {0x002ca1, 0x002ca1, 0x002ca0, 0x002ca0},
+ {0x002ca2, 0x002ca3, 0x002ca2, 0x002ca2},
+ {0x002ca3, 0x002ca3, 0x002ca2, 0x002ca2},
+ {0x002ca4, 0x002ca5, 0x002ca4, 0x002ca4},
+ {0x002ca5, 0x002ca5, 0x002ca4, 0x002ca4},
+ {0x002ca6, 0x002ca7, 0x002ca6, 0x002ca6},
+ {0x002ca7, 0x002ca7, 0x002ca6, 0x002ca6},
+ {0x002ca8, 0x002ca9, 0x002ca8, 0x002ca8},
+ {0x002ca9, 0x002ca9, 0x002ca8, 0x002ca8},
+ {0x002caa, 0x002cab, 0x002caa, 0x002caa},
+ {0x002cab, 0x002cab, 0x002caa, 0x002caa},
+ {0x002cac, 0x002cad, 0x002cac, 0x002cac},
+ {0x002cad, 0x002cad, 0x002cac, 0x002cac},
+ {0x002cae, 0x002caf, 0x002cae, 0x002cae},
+ {0x002caf, 0x002caf, 0x002cae, 0x002cae},
+ {0x002cb0, 0x002cb1, 0x002cb0, 0x002cb0},
+ {0x002cb1, 0x002cb1, 0x002cb0, 0x002cb0},
+ {0x002cb2, 0x002cb3, 0x002cb2, 0x002cb2},
+ {0x002cb3, 0x002cb3, 0x002cb2, 0x002cb2},
+ {0x002cb4, 0x002cb5, 0x002cb4, 0x002cb4},
+ {0x002cb5, 0x002cb5, 0x002cb4, 0x002cb4},
+ {0x002cb6, 0x002cb7, 0x002cb6, 0x002cb6},
+ {0x002cb7, 0x002cb7, 0x002cb6, 0x002cb6},
+ {0x002cb8, 0x002cb9, 0x002cb8, 0x002cb8},
+ {0x002cb9, 0x002cb9, 0x002cb8, 0x002cb8},
+ {0x002cba, 0x002cbb, 0x002cba, 0x002cba},
+ {0x002cbb, 0x002cbb, 0x002cba, 0x002cba},
+ {0x002cbc, 0x002cbd, 0x002cbc, 0x002cbc},
+ {0x002cbd, 0x002cbd, 0x002cbc, 0x002cbc},
+ {0x002cbe, 0x002cbf, 0x002cbe, 0x002cbe},
+ {0x002cbf, 0x002cbf, 0x002cbe, 0x002cbe},
+ {0x002cc0, 0x002cc1, 0x002cc0, 0x002cc0},
+ {0x002cc1, 0x002cc1, 0x002cc0, 0x002cc0},
+ {0x002cc2, 0x002cc3, 0x002cc2, 0x002cc2},
+ {0x002cc3, 0x002cc3, 0x002cc2, 0x002cc2},
+ {0x002cc4, 0x002cc5, 0x002cc4, 0x002cc4},
+ {0x002cc5, 0x002cc5, 0x002cc4, 0x002cc4},
+ {0x002cc6, 0x002cc7, 0x002cc6, 0x002cc6},
+ {0x002cc7, 0x002cc7, 0x002cc6, 0x002cc6},
+ {0x002cc8, 0x002cc9, 0x002cc8, 0x002cc8},
+ {0x002cc9, 0x002cc9, 0x002cc8, 0x002cc8},
+ {0x002cca, 0x002ccb, 0x002cca, 0x002cca},
+ {0x002ccb, 0x002ccb, 0x002cca, 0x002cca},
+ {0x002ccc, 0x002ccd, 0x002ccc, 0x002ccc},
+ {0x002ccd, 0x002ccd, 0x002ccc, 0x002ccc},
+ {0x002cce, 0x002ccf, 0x002cce, 0x002cce},
+ {0x002ccf, 0x002ccf, 0x002cce, 0x002cce},
+ {0x002cd0, 0x002cd1, 0x002cd0, 0x002cd0},
+ {0x002cd1, 0x002cd1, 0x002cd0, 0x002cd0},
+ {0x002cd2, 0x002cd3, 0x002cd2, 0x002cd2},
+ {0x002cd3, 0x002cd3, 0x002cd2, 0x002cd2},
+ {0x002cd4, 0x002cd5, 0x002cd4, 0x002cd4},
+ {0x002cd5, 0x002cd5, 0x002cd4, 0x002cd4},
+ {0x002cd6, 0x002cd7, 0x002cd6, 0x002cd6},
+ {0x002cd7, 0x002cd7, 0x002cd6, 0x002cd6},
+ {0x002cd8, 0x002cd9, 0x002cd8, 0x002cd8},
+ {0x002cd9, 0x002cd9, 0x002cd8, 0x002cd8},
+ {0x002cda, 0x002cdb, 0x002cda, 0x002cda},
+ {0x002cdb, 0x002cdb, 0x002cda, 0x002cda},
+ {0x002cdc, 0x002cdd, 0x002cdc, 0x002cdc},
+ {0x002cdd, 0x002cdd, 0x002cdc, 0x002cdc},
+ {0x002cde, 0x002cdf, 0x002cde, 0x002cde},
+ {0x002cdf, 0x002cdf, 0x002cde, 0x002cde},
+ {0x002ce0, 0x002ce1, 0x002ce0, 0x002ce0},
+ {0x002ce1, 0x002ce1, 0x002ce0, 0x002ce0},
+ {0x002ce2, 0x002ce3, 0x002ce2, 0x002ce2},
+ {0x002ce3, 0x002ce3, 0x002ce2, 0x002ce2},
+ {0x002ceb, 0x002cec, 0x002ceb, 0x002ceb},
+ {0x002cec, 0x002cec, 0x002ceb, 0x002ceb},
+ {0x002ced, 0x002cee, 0x002ced, 0x002ced},
+ {0x002cee, 0x002cee, 0x002ced, 0x002ced},
+ {0x002cf2, 0x002cf3, 0x002cf2, 0x002cf2},
+ {0x002cf3, 0x002cf3, 0x002cf2, 0x002cf2},
+ {0x002d00, 0x002d00, 0x0010a0, 0x0010a0},
+ {0x002d01, 0x002d01, 0x0010a1, 0x0010a1},
+ {0x002d02, 0x002d02, 0x0010a2, 0x0010a2},
+ {0x002d03, 0x002d03, 0x0010a3, 0x0010a3},
+ {0x002d04, 0x002d04, 0x0010a4, 0x0010a4},
+ {0x002d05, 0x002d05, 0x0010a5, 0x0010a5},
+ {0x002d06, 0x002d06, 0x0010a6, 0x0010a6},
+ {0x002d07, 0x002d07, 0x0010a7, 0x0010a7},
+ {0x002d08, 0x002d08, 0x0010a8, 0x0010a8},
+ {0x002d09, 0x002d09, 0x0010a9, 0x0010a9},
+ {0x002d0a, 0x002d0a, 0x0010aa, 0x0010aa},
+ {0x002d0b, 0x002d0b, 0x0010ab, 0x0010ab},
+ {0x002d0c, 0x002d0c, 0x0010ac, 0x0010ac},
+ {0x002d0d, 0x002d0d, 0x0010ad, 0x0010ad},
+ {0x002d0e, 0x002d0e, 0x0010ae, 0x0010ae},
+ {0x002d0f, 0x002d0f, 0x0010af, 0x0010af},
+ {0x002d10, 0x002d10, 0x0010b0, 0x0010b0},
+ {0x002d11, 0x002d11, 0x0010b1, 0x0010b1},
+ {0x002d12, 0x002d12, 0x0010b2, 0x0010b2},
+ {0x002d13, 0x002d13, 0x0010b3, 0x0010b3},
+ {0x002d14, 0x002d14, 0x0010b4, 0x0010b4},
+ {0x002d15, 0x002d15, 0x0010b5, 0x0010b5},
+ {0x002d16, 0x002d16, 0x0010b6, 0x0010b6},
+ {0x002d17, 0x002d17, 0x0010b7, 0x0010b7},
+ {0x002d18, 0x002d18, 0x0010b8, 0x0010b8},
+ {0x002d19, 0x002d19, 0x0010b9, 0x0010b9},
+ {0x002d1a, 0x002d1a, 0x0010ba, 0x0010ba},
+ {0x002d1b, 0x002d1b, 0x0010bb, 0x0010bb},
+ {0x002d1c, 0x002d1c, 0x0010bc, 0x0010bc},
+ {0x002d1d, 0x002d1d, 0x0010bd, 0x0010bd},
+ {0x002d1e, 0x002d1e, 0x0010be, 0x0010be},
+ {0x002d1f, 0x002d1f, 0x0010bf, 0x0010bf},
+ {0x002d20, 0x002d20, 0x0010c0, 0x0010c0},
+ {0x002d21, 0x002d21, 0x0010c1, 0x0010c1},
+ {0x002d22, 0x002d22, 0x0010c2, 0x0010c2},
+ {0x002d23, 0x002d23, 0x0010c3, 0x0010c3},
+ {0x002d24, 0x002d24, 0x0010c4, 0x0010c4},
+ {0x002d25, 0x002d25, 0x0010c5, 0x0010c5},
+ {0x002d27, 0x002d27, 0x0010c7, 0x0010c7},
+ {0x002d2d, 0x002d2d, 0x0010cd, 0x0010cd},
+ {0x00a640, 0x00a641, 0x00a640, 0x00a640},
+ {0x00a641, 0x00a641, 0x00a640, 0x00a640},
+ {0x00a642, 0x00a643, 0x00a642, 0x00a642},
+ {0x00a643, 0x00a643, 0x00a642, 0x00a642},
+ {0x00a644, 0x00a645, 0x00a644, 0x00a644},
+ {0x00a645, 0x00a645, 0x00a644, 0x00a644},
+ {0x00a646, 0x00a647, 0x00a646, 0x00a646},
+ {0x00a647, 0x00a647, 0x00a646, 0x00a646},
+ {0x00a648, 0x00a649, 0x00a648, 0x00a648},
+ {0x00a649, 0x00a649, 0x00a648, 0x00a648},
+ {0x00a64a, 0x00a64b, 0x00a64a, 0x00a64a},
+ {0x00a64b, 0x00a64b, 0x00a64a, 0x00a64a},
+ {0x00a64c, 0x00a64d, 0x00a64c, 0x00a64c},
+ {0x00a64d, 0x00a64d, 0x00a64c, 0x00a64c},
+ {0x00a64e, 0x00a64f, 0x00a64e, 0x00a64e},
+ {0x00a64f, 0x00a64f, 0x00a64e, 0x00a64e},
+ {0x00a650, 0x00a651, 0x00a650, 0x00a650},
+ {0x00a651, 0x00a651, 0x00a650, 0x00a650},
+ {0x00a652, 0x00a653, 0x00a652, 0x00a652},
+ {0x00a653, 0x00a653, 0x00a652, 0x00a652},
+ {0x00a654, 0x00a655, 0x00a654, 0x00a654},
+ {0x00a655, 0x00a655, 0x00a654, 0x00a654},
+ {0x00a656, 0x00a657, 0x00a656, 0x00a656},
+ {0x00a657, 0x00a657, 0x00a656, 0x00a656},
+ {0x00a658, 0x00a659, 0x00a658, 0x00a658},
+ {0x00a659, 0x00a659, 0x00a658, 0x00a658},
+ {0x00a65a, 0x00a65b, 0x00a65a, 0x00a65a},
+ {0x00a65b, 0x00a65b, 0x00a65a, 0x00a65a},
+ {0x00a65c, 0x00a65d, 0x00a65c, 0x00a65c},
+ {0x00a65d, 0x00a65d, 0x00a65c, 0x00a65c},
+ {0x00a65e, 0x00a65f, 0x00a65e, 0x00a65e},
+ {0x00a65f, 0x00a65f, 0x00a65e, 0x00a65e},
+ {0x00a660, 0x00a661, 0x00a660, 0x00a660},
+ {0x00a661, 0x00a661, 0x00a660, 0x00a660},
+ {0x00a662, 0x00a663, 0x00a662, 0x00a662},
+ {0x00a663, 0x00a663, 0x00a662, 0x00a662},
+ {0x00a664, 0x00a665, 0x00a664, 0x00a664},
+ {0x00a665, 0x00a665, 0x00a664, 0x00a664},
+ {0x00a666, 0x00a667, 0x00a666, 0x00a666},
+ {0x00a667, 0x00a667, 0x00a666, 0x00a666},
+ {0x00a668, 0x00a669, 0x00a668, 0x00a668},
+ {0x00a669, 0x00a669, 0x00a668, 0x00a668},
+ {0x00a66a, 0x00a66b, 0x00a66a, 0x00a66a},
+ {0x00a66b, 0x00a66b, 0x00a66a, 0x00a66a},
+ {0x00a66c, 0x00a66d, 0x00a66c, 0x00a66c},
+ {0x00a66d, 0x00a66d, 0x00a66c, 0x00a66c},
+ {0x00a680, 0x00a681, 0x00a680, 0x00a680},
+ {0x00a681, 0x00a681, 0x00a680, 0x00a680},
+ {0x00a682, 0x00a683, 0x00a682, 0x00a682},
+ {0x00a683, 0x00a683, 0x00a682, 0x00a682},
+ {0x00a684, 0x00a685, 0x00a684, 0x00a684},
+ {0x00a685, 0x00a685, 0x00a684, 0x00a684},
+ {0x00a686, 0x00a687, 0x00a686, 0x00a686},
+ {0x00a687, 0x00a687, 0x00a686, 0x00a686},
+ {0x00a688, 0x00a689, 0x00a688, 0x00a688},
+ {0x00a689, 0x00a689, 0x00a688, 0x00a688},
+ {0x00a68a, 0x00a68b, 0x00a68a, 0x00a68a},
+ {0x00a68b, 0x00a68b, 0x00a68a, 0x00a68a},
+ {0x00a68c, 0x00a68d, 0x00a68c, 0x00a68c},
+ {0x00a68d, 0x00a68d, 0x00a68c, 0x00a68c},
+ {0x00a68e, 0x00a68f, 0x00a68e, 0x00a68e},
+ {0x00a68f, 0x00a68f, 0x00a68e, 0x00a68e},
+ {0x00a690, 0x00a691, 0x00a690, 0x00a690},
+ {0x00a691, 0x00a691, 0x00a690, 0x00a690},
+ {0x00a692, 0x00a693, 0x00a692, 0x00a692},
+ {0x00a693, 0x00a693, 0x00a692, 0x00a692},
+ {0x00a694, 0x00a695, 0x00a694, 0x00a694},
+ {0x00a695, 0x00a695, 0x00a694, 0x00a694},
+ {0x00a696, 0x00a697, 0x00a696, 0x00a696},
+ {0x00a697, 0x00a697, 0x00a696, 0x00a696},
+ {0x00a698, 0x00a699, 0x00a698, 0x00a698},
+ {0x00a699, 0x00a699, 0x00a698, 0x00a698},
+ {0x00a69a, 0x00a69b, 0x00a69a, 0x00a69a},
+ {0x00a69b, 0x00a69b, 0x00a69a, 0x00a69a},
+ {0x00a722, 0x00a723, 0x00a722, 0x00a722},
+ {0x00a723, 0x00a723, 0x00a722, 0x00a722},
+ {0x00a724, 0x00a725, 0x00a724, 0x00a724},
+ {0x00a725, 0x00a725, 0x00a724, 0x00a724},
+ {0x00a726, 0x00a727, 0x00a726, 0x00a726},
+ {0x00a727, 0x00a727, 0x00a726, 0x00a726},
+ {0x00a728, 0x00a729, 0x00a728, 0x00a728},
+ {0x00a729, 0x00a729, 0x00a728, 0x00a728},
+ {0x00a72a, 0x00a72b, 0x00a72a, 0x00a72a},
+ {0x00a72b, 0x00a72b, 0x00a72a, 0x00a72a},
+ {0x00a72c, 0x00a72d, 0x00a72c, 0x00a72c},
+ {0x00a72d, 0x00a72d, 0x00a72c, 0x00a72c},
+ {0x00a72e, 0x00a72f, 0x00a72e, 0x00a72e},
+ {0x00a72f, 0x00a72f, 0x00a72e, 0x00a72e},
+ {0x00a732, 0x00a733, 0x00a732, 0x00a732},
+ {0x00a733, 0x00a733, 0x00a732, 0x00a732},
+ {0x00a734, 0x00a735, 0x00a734, 0x00a734},
+ {0x00a735, 0x00a735, 0x00a734, 0x00a734},
+ {0x00a736, 0x00a737, 0x00a736, 0x00a736},
+ {0x00a737, 0x00a737, 0x00a736, 0x00a736},
+ {0x00a738, 0x00a739, 0x00a738, 0x00a738},
+ {0x00a739, 0x00a739, 0x00a738, 0x00a738},
+ {0x00a73a, 0x00a73b, 0x00a73a, 0x00a73a},
+ {0x00a73b, 0x00a73b, 0x00a73a, 0x00a73a},
+ {0x00a73c, 0x00a73d, 0x00a73c, 0x00a73c},
+ {0x00a73d, 0x00a73d, 0x00a73c, 0x00a73c},
+ {0x00a73e, 0x00a73f, 0x00a73e, 0x00a73e},
+ {0x00a73f, 0x00a73f, 0x00a73e, 0x00a73e},
+ {0x00a740, 0x00a741, 0x00a740, 0x00a740},
+ {0x00a741, 0x00a741, 0x00a740, 0x00a740},
+ {0x00a742, 0x00a743, 0x00a742, 0x00a742},
+ {0x00a743, 0x00a743, 0x00a742, 0x00a742},
+ {0x00a744, 0x00a745, 0x00a744, 0x00a744},
+ {0x00a745, 0x00a745, 0x00a744, 0x00a744},
+ {0x00a746, 0x00a747, 0x00a746, 0x00a746},
+ {0x00a747, 0x00a747, 0x00a746, 0x00a746},
+ {0x00a748, 0x00a749, 0x00a748, 0x00a748},
+ {0x00a749, 0x00a749, 0x00a748, 0x00a748},
+ {0x00a74a, 0x00a74b, 0x00a74a, 0x00a74a},
+ {0x00a74b, 0x00a74b, 0x00a74a, 0x00a74a},
+ {0x00a74c, 0x00a74d, 0x00a74c, 0x00a74c},
+ {0x00a74d, 0x00a74d, 0x00a74c, 0x00a74c},
+ {0x00a74e, 0x00a74f, 0x00a74e, 0x00a74e},
+ {0x00a74f, 0x00a74f, 0x00a74e, 0x00a74e},
+ {0x00a750, 0x00a751, 0x00a750, 0x00a750},
+ {0x00a751, 0x00a751, 0x00a750, 0x00a750},
+ {0x00a752, 0x00a753, 0x00a752, 0x00a752},
+ {0x00a753, 0x00a753, 0x00a752, 0x00a752},
+ {0x00a754, 0x00a755, 0x00a754, 0x00a754},
+ {0x00a755, 0x00a755, 0x00a754, 0x00a754},
+ {0x00a756, 0x00a757, 0x00a756, 0x00a756},
+ {0x00a757, 0x00a757, 0x00a756, 0x00a756},
+ {0x00a758, 0x00a759, 0x00a758, 0x00a758},
+ {0x00a759, 0x00a759, 0x00a758, 0x00a758},
+ {0x00a75a, 0x00a75b, 0x00a75a, 0x00a75a},
+ {0x00a75b, 0x00a75b, 0x00a75a, 0x00a75a},
+ {0x00a75c, 0x00a75d, 0x00a75c, 0x00a75c},
+ {0x00a75d, 0x00a75d, 0x00a75c, 0x00a75c},
+ {0x00a75e, 0x00a75f, 0x00a75e, 0x00a75e},
+ {0x00a75f, 0x00a75f, 0x00a75e, 0x00a75e},
+ {0x00a760, 0x00a761, 0x00a760, 0x00a760},
+ {0x00a761, 0x00a761, 0x00a760, 0x00a760},
+ {0x00a762, 0x00a763, 0x00a762, 0x00a762},
+ {0x00a763, 0x00a763, 0x00a762, 0x00a762},
+ {0x00a764, 0x00a765, 0x00a764, 0x00a764},
+ {0x00a765, 0x00a765, 0x00a764, 0x00a764},
+ {0x00a766, 0x00a767, 0x00a766, 0x00a766},
+ {0x00a767, 0x00a767, 0x00a766, 0x00a766},
+ {0x00a768, 0x00a769, 0x00a768, 0x00a768},
+ {0x00a769, 0x00a769, 0x00a768, 0x00a768},
+ {0x00a76a, 0x00a76b, 0x00a76a, 0x00a76a},
+ {0x00a76b, 0x00a76b, 0x00a76a, 0x00a76a},
+ {0x00a76c, 0x00a76d, 0x00a76c, 0x00a76c},
+ {0x00a76d, 0x00a76d, 0x00a76c, 0x00a76c},
+ {0x00a76e, 0x00a76f, 0x00a76e, 0x00a76e},
+ {0x00a76f, 0x00a76f, 0x00a76e, 0x00a76e},
+ {0x00a779, 0x00a77a, 0x00a779, 0x00a779},
+ {0x00a77a, 0x00a77a, 0x00a779, 0x00a779},
+ {0x00a77b, 0x00a77c, 0x00a77b, 0x00a77b},
+ {0x00a77c, 0x00a77c, 0x00a77b, 0x00a77b},
+ {0x00a77d, 0x001d79, 0x00a77d, 0x00a77d},
+ {0x00a77e, 0x00a77f, 0x00a77e, 0x00a77e},
+ {0x00a77f, 0x00a77f, 0x00a77e, 0x00a77e},
+ {0x00a780, 0x00a781, 0x00a780, 0x00a780},
+ {0x00a781, 0x00a781, 0x00a780, 0x00a780},
+ {0x00a782, 0x00a783, 0x00a782, 0x00a782},
+ {0x00a783, 0x00a783, 0x00a782, 0x00a782},
+ {0x00a784, 0x00a785, 0x00a784, 0x00a784},
+ {0x00a785, 0x00a785, 0x00a784, 0x00a784},
+ {0x00a786, 0x00a787, 0x00a786, 0x00a786},
+ {0x00a787, 0x00a787, 0x00a786, 0x00a786},
+ {0x00a78b, 0x00a78c, 0x00a78b, 0x00a78b},
+ {0x00a78c, 0x00a78c, 0x00a78b, 0x00a78b},
+ {0x00a78d, 0x000265, 0x00a78d, 0x00a78d},
+ {0x00a790, 0x00a791, 0x00a790, 0x00a790},
+ {0x00a791, 0x00a791, 0x00a790, 0x00a790},
+ {0x00a792, 0x00a793, 0x00a792, 0x00a792},
+ {0x00a793, 0x00a793, 0x00a792, 0x00a792},
+ {0x00a794, 0x00a794, 0x00a7c4, 0x00a7c4},
+ {0x00a796, 0x00a797, 0x00a796, 0x00a796},
+ {0x00a797, 0x00a797, 0x00a796, 0x00a796},
+ {0x00a798, 0x00a799, 0x00a798, 0x00a798},
+ {0x00a799, 0x00a799, 0x00a798, 0x00a798},
+ {0x00a79a, 0x00a79b, 0x00a79a, 0x00a79a},
+ {0x00a79b, 0x00a79b, 0x00a79a, 0x00a79a},
+ {0x00a79c, 0x00a79d, 0x00a79c, 0x00a79c},
+ {0x00a79d, 0x00a79d, 0x00a79c, 0x00a79c},
+ {0x00a79e, 0x00a79f, 0x00a79e, 0x00a79e},
+ {0x00a79f, 0x00a79f, 0x00a79e, 0x00a79e},
+ {0x00a7a0, 0x00a7a1, 0x00a7a0, 0x00a7a0},
+ {0x00a7a1, 0x00a7a1, 0x00a7a0, 0x00a7a0},
+ {0x00a7a2, 0x00a7a3, 0x00a7a2, 0x00a7a2},
+ {0x00a7a3, 0x00a7a3, 0x00a7a2, 0x00a7a2},
+ {0x00a7a4, 0x00a7a5, 0x00a7a4, 0x00a7a4},
+ {0x00a7a5, 0x00a7a5, 0x00a7a4, 0x00a7a4},
+ {0x00a7a6, 0x00a7a7, 0x00a7a6, 0x00a7a6},
+ {0x00a7a7, 0x00a7a7, 0x00a7a6, 0x00a7a6},
+ {0x00a7a8, 0x00a7a9, 0x00a7a8, 0x00a7a8},
+ {0x00a7a9, 0x00a7a9, 0x00a7a8, 0x00a7a8},
+ {0x00a7aa, 0x000266, 0x00a7aa, 0x00a7aa},
+ {0x00a7ab, 0x00025c, 0x00a7ab, 0x00a7ab},
+ {0x00a7ac, 0x000261, 0x00a7ac, 0x00a7ac},
+ {0x00a7ad, 0x00026c, 0x00a7ad, 0x00a7ad},
+ {0x00a7ae, 0x00026a, 0x00a7ae, 0x00a7ae},
+ {0x00a7b0, 0x00029e, 0x00a7b0, 0x00a7b0},
+ {0x00a7b1, 0x000287, 0x00a7b1, 0x00a7b1},
+ {0x00a7b2, 0x00029d, 0x00a7b2, 0x00a7b2},
+ {0x00a7b3, 0x00ab53, 0x00a7b3, 0x00a7b3},
+ {0x00a7b4, 0x00a7b5, 0x00a7b4, 0x00a7b4},
+ {0x00a7b5, 0x00a7b5, 0x00a7b4, 0x00a7b4},
+ {0x00a7b6, 0x00a7b7, 0x00a7b6, 0x00a7b6},
+ {0x00a7b7, 0x00a7b7, 0x00a7b6, 0x00a7b6},
+ {0x00a7b8, 0x00a7b9, 0x00a7b8, 0x00a7b8},
+ {0x00a7b9, 0x00a7b9, 0x00a7b8, 0x00a7b8},
+ {0x00a7ba, 0x00a7bb, 0x00a7ba, 0x00a7ba},
+ {0x00a7bb, 0x00a7bb, 0x00a7ba, 0x00a7ba},
+ {0x00a7bc, 0x00a7bd, 0x00a7bc, 0x00a7bc},
+ {0x00a7bd, 0x00a7bd, 0x00a7bc, 0x00a7bc},
+ {0x00a7be, 0x00a7bf, 0x00a7be, 0x00a7be},
+ {0x00a7bf, 0x00a7bf, 0x00a7be, 0x00a7be},
+ {0x00a7c0, 0x00a7c1, 0x00a7c0, 0x00a7c0},
+ {0x00a7c1, 0x00a7c1, 0x00a7c0, 0x00a7c0},
+ {0x00a7c2, 0x00a7c3, 0x00a7c2, 0x00a7c2},
+ {0x00a7c3, 0x00a7c3, 0x00a7c2, 0x00a7c2},
+ {0x00a7c4, 0x00a794, 0x00a7c4, 0x00a7c4},
+ {0x00a7c5, 0x000282, 0x00a7c5, 0x00a7c5},
+ {0x00a7c6, 0x001d8e, 0x00a7c6, 0x00a7c6},
+ {0x00a7c7, 0x00a7c8, 0x00a7c7, 0x00a7c7},
+ {0x00a7c8, 0x00a7c8, 0x00a7c7, 0x00a7c7},
+ {0x00a7c9, 0x00a7ca, 0x00a7c9, 0x00a7c9},
+ {0x00a7ca, 0x00a7ca, 0x00a7c9, 0x00a7c9},
+ {0x00a7d0, 0x00a7d1, 0x00a7d0, 0x00a7d0},
+ {0x00a7d1, 0x00a7d1, 0x00a7d0, 0x00a7d0},
+ {0x00a7d6, 0x00a7d7, 0x00a7d6, 0x00a7d6},
+ {0x00a7d7, 0x00a7d7, 0x00a7d6, 0x00a7d6},
+ {0x00a7d8, 0x00a7d9, 0x00a7d8, 0x00a7d8},
+ {0x00a7d9, 0x00a7d9, 0x00a7d8, 0x00a7d8},
+ {0x00a7f5, 0x00a7f6, 0x00a7f5, 0x00a7f5},
+ {0x00a7f6, 0x00a7f6, 0x00a7f5, 0x00a7f5},
+ {0x00ab53, 0x00ab53, 0x00a7b3, 0x00a7b3},
+ {0x00ab70, 0x00ab70, 0x0013a0, 0x0013a0},
+ {0x00ab71, 0x00ab71, 0x0013a1, 0x0013a1},
+ {0x00ab72, 0x00ab72, 0x0013a2, 0x0013a2},
+ {0x00ab73, 0x00ab73, 0x0013a3, 0x0013a3},
+ {0x00ab74, 0x00ab74, 0x0013a4, 0x0013a4},
+ {0x00ab75, 0x00ab75, 0x0013a5, 0x0013a5},
+ {0x00ab76, 0x00ab76, 0x0013a6, 0x0013a6},
+ {0x00ab77, 0x00ab77, 0x0013a7, 0x0013a7},
+ {0x00ab78, 0x00ab78, 0x0013a8, 0x0013a8},
+ {0x00ab79, 0x00ab79, 0x0013a9, 0x0013a9},
+ {0x00ab7a, 0x00ab7a, 0x0013aa, 0x0013aa},
+ {0x00ab7b, 0x00ab7b, 0x0013ab, 0x0013ab},
+ {0x00ab7c, 0x00ab7c, 0x0013ac, 0x0013ac},
+ {0x00ab7d, 0x00ab7d, 0x0013ad, 0x0013ad},
+ {0x00ab7e, 0x00ab7e, 0x0013ae, 0x0013ae},
+ {0x00ab7f, 0x00ab7f, 0x0013af, 0x0013af},
+ {0x00ab80, 0x00ab80, 0x0013b0, 0x0013b0},
+ {0x00ab81, 0x00ab81, 0x0013b1, 0x0013b1},
+ {0x00ab82, 0x00ab82, 0x0013b2, 0x0013b2},
+ {0x00ab83, 0x00ab83, 0x0013b3, 0x0013b3},
+ {0x00ab84, 0x00ab84, 0x0013b4, 0x0013b4},
+ {0x00ab85, 0x00ab85, 0x0013b5, 0x0013b5},
+ {0x00ab86, 0x00ab86, 0x0013b6, 0x0013b6},
+ {0x00ab87, 0x00ab87, 0x0013b7, 0x0013b7},
+ {0x00ab88, 0x00ab88, 0x0013b8, 0x0013b8},
+ {0x00ab89, 0x00ab89, 0x0013b9, 0x0013b9},
+ {0x00ab8a, 0x00ab8a, 0x0013ba, 0x0013ba},
+ {0x00ab8b, 0x00ab8b, 0x0013bb, 0x0013bb},
+ {0x00ab8c, 0x00ab8c, 0x0013bc, 0x0013bc},
+ {0x00ab8d, 0x00ab8d, 0x0013bd, 0x0013bd},
+ {0x00ab8e, 0x00ab8e, 0x0013be, 0x0013be},
+ {0x00ab8f, 0x00ab8f, 0x0013bf, 0x0013bf},
+ {0x00ab90, 0x00ab90, 0x0013c0, 0x0013c0},
+ {0x00ab91, 0x00ab91, 0x0013c1, 0x0013c1},
+ {0x00ab92, 0x00ab92, 0x0013c2, 0x0013c2},
+ {0x00ab93, 0x00ab93, 0x0013c3, 0x0013c3},
+ {0x00ab94, 0x00ab94, 0x0013c4, 0x0013c4},
+ {0x00ab95, 0x00ab95, 0x0013c5, 0x0013c5},
+ {0x00ab96, 0x00ab96, 0x0013c6, 0x0013c6},
+ {0x00ab97, 0x00ab97, 0x0013c7, 0x0013c7},
+ {0x00ab98, 0x00ab98, 0x0013c8, 0x0013c8},
+ {0x00ab99, 0x00ab99, 0x0013c9, 0x0013c9},
+ {0x00ab9a, 0x00ab9a, 0x0013ca, 0x0013ca},
+ {0x00ab9b, 0x00ab9b, 0x0013cb, 0x0013cb},
+ {0x00ab9c, 0x00ab9c, 0x0013cc, 0x0013cc},
+ {0x00ab9d, 0x00ab9d, 0x0013cd, 0x0013cd},
+ {0x00ab9e, 0x00ab9e, 0x0013ce, 0x0013ce},
+ {0x00ab9f, 0x00ab9f, 0x0013cf, 0x0013cf},
+ {0x00aba0, 0x00aba0, 0x0013d0, 0x0013d0},
+ {0x00aba1, 0x00aba1, 0x0013d1, 0x0013d1},
+ {0x00aba2, 0x00aba2, 0x0013d2, 0x0013d2},
+ {0x00aba3, 0x00aba3, 0x0013d3, 0x0013d3},
+ {0x00aba4, 0x00aba4, 0x0013d4, 0x0013d4},
+ {0x00aba5, 0x00aba5, 0x0013d5, 0x0013d5},
+ {0x00aba6, 0x00aba6, 0x0013d6, 0x0013d6},
+ {0x00aba7, 0x00aba7, 0x0013d7, 0x0013d7},
+ {0x00aba8, 0x00aba8, 0x0013d8, 0x0013d8},
+ {0x00aba9, 0x00aba9, 0x0013d9, 0x0013d9},
+ {0x00abaa, 0x00abaa, 0x0013da, 0x0013da},
+ {0x00abab, 0x00abab, 0x0013db, 0x0013db},
+ {0x00abac, 0x00abac, 0x0013dc, 0x0013dc},
+ {0x00abad, 0x00abad, 0x0013dd, 0x0013dd},
+ {0x00abae, 0x00abae, 0x0013de, 0x0013de},
+ {0x00abaf, 0x00abaf, 0x0013df, 0x0013df},
+ {0x00abb0, 0x00abb0, 0x0013e0, 0x0013e0},
+ {0x00abb1, 0x00abb1, 0x0013e1, 0x0013e1},
+ {0x00abb2, 0x00abb2, 0x0013e2, 0x0013e2},
+ {0x00abb3, 0x00abb3, 0x0013e3, 0x0013e3},
+ {0x00abb4, 0x00abb4, 0x0013e4, 0x0013e4},
+ {0x00abb5, 0x00abb5, 0x0013e5, 0x0013e5},
+ {0x00abb6, 0x00abb6, 0x0013e6, 0x0013e6},
+ {0x00abb7, 0x00abb7, 0x0013e7, 0x0013e7},
+ {0x00abb8, 0x00abb8, 0x0013e8, 0x0013e8},
+ {0x00abb9, 0x00abb9, 0x0013e9, 0x0013e9},
+ {0x00abba, 0x00abba, 0x0013ea, 0x0013ea},
+ {0x00abbb, 0x00abbb, 0x0013eb, 0x0013eb},
+ {0x00abbc, 0x00abbc, 0x0013ec, 0x0013ec},
+ {0x00abbd, 0x00abbd, 0x0013ed, 0x0013ed},
+ {0x00abbe, 0x00abbe, 0x0013ee, 0x0013ee},
+ {0x00abbf, 0x00abbf, 0x0013ef, 0x0013ef},
+ {0x00ff21, 0x00ff41, 0x00ff21, 0x00ff21},
+ {0x00ff22, 0x00ff42, 0x00ff22, 0x00ff22},
+ {0x00ff23, 0x00ff43, 0x00ff23, 0x00ff23},
+ {0x00ff24, 0x00ff44, 0x00ff24, 0x00ff24},
+ {0x00ff25, 0x00ff45, 0x00ff25, 0x00ff25},
+ {0x00ff26, 0x00ff46, 0x00ff26, 0x00ff26},
+ {0x00ff27, 0x00ff47, 0x00ff27, 0x00ff27},
+ {0x00ff28, 0x00ff48, 0x00ff28, 0x00ff28},
+ {0x00ff29, 0x00ff49, 0x00ff29, 0x00ff29},
+ {0x00ff2a, 0x00ff4a, 0x00ff2a, 0x00ff2a},
+ {0x00ff2b, 0x00ff4b, 0x00ff2b, 0x00ff2b},
+ {0x00ff2c, 0x00ff4c, 0x00ff2c, 0x00ff2c},
+ {0x00ff2d, 0x00ff4d, 0x00ff2d, 0x00ff2d},
+ {0x00ff2e, 0x00ff4e, 0x00ff2e, 0x00ff2e},
+ {0x00ff2f, 0x00ff4f, 0x00ff2f, 0x00ff2f},
+ {0x00ff30, 0x00ff50, 0x00ff30, 0x00ff30},
+ {0x00ff31, 0x00ff51, 0x00ff31, 0x00ff31},
+ {0x00ff32, 0x00ff52, 0x00ff32, 0x00ff32},
+ {0x00ff33, 0x00ff53, 0x00ff33, 0x00ff33},
+ {0x00ff34, 0x00ff54, 0x00ff34, 0x00ff34},
+ {0x00ff35, 0x00ff55, 0x00ff35, 0x00ff35},
+ {0x00ff36, 0x00ff56, 0x00ff36, 0x00ff36},
+ {0x00ff37, 0x00ff57, 0x00ff37, 0x00ff37},
+ {0x00ff38, 0x00ff58, 0x00ff38, 0x00ff38},
+ {0x00ff39, 0x00ff59, 0x00ff39, 0x00ff39},
+ {0x00ff3a, 0x00ff5a, 0x00ff3a, 0x00ff3a},
+ {0x00ff41, 0x00ff41, 0x00ff21, 0x00ff21},
+ {0x00ff42, 0x00ff42, 0x00ff22, 0x00ff22},
+ {0x00ff43, 0x00ff43, 0x00ff23, 0x00ff23},
+ {0x00ff44, 0x00ff44, 0x00ff24, 0x00ff24},
+ {0x00ff45, 0x00ff45, 0x00ff25, 0x00ff25},
+ {0x00ff46, 0x00ff46, 0x00ff26, 0x00ff26},
+ {0x00ff47, 0x00ff47, 0x00ff27, 0x00ff27},
+ {0x00ff48, 0x00ff48, 0x00ff28, 0x00ff28},
+ {0x00ff49, 0x00ff49, 0x00ff29, 0x00ff29},
+ {0x00ff4a, 0x00ff4a, 0x00ff2a, 0x00ff2a},
+ {0x00ff4b, 0x00ff4b, 0x00ff2b, 0x00ff2b},
+ {0x00ff4c, 0x00ff4c, 0x00ff2c, 0x00ff2c},
+ {0x00ff4d, 0x00ff4d, 0x00ff2d, 0x00ff2d},
+ {0x00ff4e, 0x00ff4e, 0x00ff2e, 0x00ff2e},
+ {0x00ff4f, 0x00ff4f, 0x00ff2f, 0x00ff2f},
+ {0x00ff50, 0x00ff50, 0x00ff30, 0x00ff30},
+ {0x00ff51, 0x00ff51, 0x00ff31, 0x00ff31},
+ {0x00ff52, 0x00ff52, 0x00ff32, 0x00ff32},
+ {0x00ff53, 0x00ff53, 0x00ff33, 0x00ff33},
+ {0x00ff54, 0x00ff54, 0x00ff34, 0x00ff34},
+ {0x00ff55, 0x00ff55, 0x00ff35, 0x00ff35},
+ {0x00ff56, 0x00ff56, 0x00ff36, 0x00ff36},
+ {0x00ff57, 0x00ff57, 0x00ff37, 0x00ff37},
+ {0x00ff58, 0x00ff58, 0x00ff38, 0x00ff38},
+ {0x00ff59, 0x00ff59, 0x00ff39, 0x00ff39},
+ {0x00ff5a, 0x00ff5a, 0x00ff3a, 0x00ff3a},
+ {0x010400, 0x010428, 0x010400, 0x010400},
+ {0x010401, 0x010429, 0x010401, 0x010401},
+ {0x010402, 0x01042a, 0x010402, 0x010402},
+ {0x010403, 0x01042b, 0x010403, 0x010403},
+ {0x010404, 0x01042c, 0x010404, 0x010404},
+ {0x010405, 0x01042d, 0x010405, 0x010405},
+ {0x010406, 0x01042e, 0x010406, 0x010406},
+ {0x010407, 0x01042f, 0x010407, 0x010407},
+ {0x010408, 0x010430, 0x010408, 0x010408},
+ {0x010409, 0x010431, 0x010409, 0x010409},
+ {0x01040a, 0x010432, 0x01040a, 0x01040a},
+ {0x01040b, 0x010433, 0x01040b, 0x01040b},
+ {0x01040c, 0x010434, 0x01040c, 0x01040c},
+ {0x01040d, 0x010435, 0x01040d, 0x01040d},
+ {0x01040e, 0x010436, 0x01040e, 0x01040e},
+ {0x01040f, 0x010437, 0x01040f, 0x01040f},
+ {0x010410, 0x010438, 0x010410, 0x010410},
+ {0x010411, 0x010439, 0x010411, 0x010411},
+ {0x010412, 0x01043a, 0x010412, 0x010412},
+ {0x010413, 0x01043b, 0x010413, 0x010413},
+ {0x010414, 0x01043c, 0x010414, 0x010414},
+ {0x010415, 0x01043d, 0x010415, 0x010415},
+ {0x010416, 0x01043e, 0x010416, 0x010416},
+ {0x010417, 0x01043f, 0x010417, 0x010417},
+ {0x010418, 0x010440, 0x010418, 0x010418},
+ {0x010419, 0x010441, 0x010419, 0x010419},
+ {0x01041a, 0x010442, 0x01041a, 0x01041a},
+ {0x01041b, 0x010443, 0x01041b, 0x01041b},
+ {0x01041c, 0x010444, 0x01041c, 0x01041c},
+ {0x01041d, 0x010445, 0x01041d, 0x01041d},
+ {0x01041e, 0x010446, 0x01041e, 0x01041e},
+ {0x01041f, 0x010447, 0x01041f, 0x01041f},
+ {0x010420, 0x010448, 0x010420, 0x010420},
+ {0x010421, 0x010449, 0x010421, 0x010421},
+ {0x010422, 0x01044a, 0x010422, 0x010422},
+ {0x010423, 0x01044b, 0x010423, 0x010423},
+ {0x010424, 0x01044c, 0x010424, 0x010424},
+ {0x010425, 0x01044d, 0x010425, 0x010425},
+ {0x010426, 0x01044e, 0x010426, 0x010426},
+ {0x010427, 0x01044f, 0x010427, 0x010427},
+ {0x010428, 0x010428, 0x010400, 0x010400},
+ {0x010429, 0x010429, 0x010401, 0x010401},
+ {0x01042a, 0x01042a, 0x010402, 0x010402},
+ {0x01042b, 0x01042b, 0x010403, 0x010403},
+ {0x01042c, 0x01042c, 0x010404, 0x010404},
+ {0x01042d, 0x01042d, 0x010405, 0x010405},
+ {0x01042e, 0x01042e, 0x010406, 0x010406},
+ {0x01042f, 0x01042f, 0x010407, 0x010407},
+ {0x010430, 0x010430, 0x010408, 0x010408},
+ {0x010431, 0x010431, 0x010409, 0x010409},
+ {0x010432, 0x010432, 0x01040a, 0x01040a},
+ {0x010433, 0x010433, 0x01040b, 0x01040b},
+ {0x010434, 0x010434, 0x01040c, 0x01040c},
+ {0x010435, 0x010435, 0x01040d, 0x01040d},
+ {0x010436, 0x010436, 0x01040e, 0x01040e},
+ {0x010437, 0x010437, 0x01040f, 0x01040f},
+ {0x010438, 0x010438, 0x010410, 0x010410},
+ {0x010439, 0x010439, 0x010411, 0x010411},
+ {0x01043a, 0x01043a, 0x010412, 0x010412},
+ {0x01043b, 0x01043b, 0x010413, 0x010413},
+ {0x01043c, 0x01043c, 0x010414, 0x010414},
+ {0x01043d, 0x01043d, 0x010415, 0x010415},
+ {0x01043e, 0x01043e, 0x010416, 0x010416},
+ {0x01043f, 0x01043f, 0x010417, 0x010417},
+ {0x010440, 0x010440, 0x010418, 0x010418},
+ {0x010441, 0x010441, 0x010419, 0x010419},
+ {0x010442, 0x010442, 0x01041a, 0x01041a},
+ {0x010443, 0x010443, 0x01041b, 0x01041b},
+ {0x010444, 0x010444, 0x01041c, 0x01041c},
+ {0x010445, 0x010445, 0x01041d, 0x01041d},
+ {0x010446, 0x010446, 0x01041e, 0x01041e},
+ {0x010447, 0x010447, 0x01041f, 0x01041f},
+ {0x010448, 0x010448, 0x010420, 0x010420},
+ {0x010449, 0x010449, 0x010421, 0x010421},
+ {0x01044a, 0x01044a, 0x010422, 0x010422},
+ {0x01044b, 0x01044b, 0x010423, 0x010423},
+ {0x01044c, 0x01044c, 0x010424, 0x010424},
+ {0x01044d, 0x01044d, 0x010425, 0x010425},
+ {0x01044e, 0x01044e, 0x010426, 0x010426},
+ {0x01044f, 0x01044f, 0x010427, 0x010427},
+ {0x0104b0, 0x0104d8, 0x0104b0, 0x0104b0},
+ {0x0104b1, 0x0104d9, 0x0104b1, 0x0104b1},
+ {0x0104b2, 0x0104da, 0x0104b2, 0x0104b2},
+ {0x0104b3, 0x0104db, 0x0104b3, 0x0104b3},
+ {0x0104b4, 0x0104dc, 0x0104b4, 0x0104b4},
+ {0x0104b5, 0x0104dd, 0x0104b5, 0x0104b5},
+ {0x0104b6, 0x0104de, 0x0104b6, 0x0104b6},
+ {0x0104b7, 0x0104df, 0x0104b7, 0x0104b7},
+ {0x0104b8, 0x0104e0, 0x0104b8, 0x0104b8},
+ {0x0104b9, 0x0104e1, 0x0104b9, 0x0104b9},
+ {0x0104ba, 0x0104e2, 0x0104ba, 0x0104ba},
+ {0x0104bb, 0x0104e3, 0x0104bb, 0x0104bb},
+ {0x0104bc, 0x0104e4, 0x0104bc, 0x0104bc},
+ {0x0104bd, 0x0104e5, 0x0104bd, 0x0104bd},
+ {0x0104be, 0x0104e6, 0x0104be, 0x0104be},
+ {0x0104bf, 0x0104e7, 0x0104bf, 0x0104bf},
+ {0x0104c0, 0x0104e8, 0x0104c0, 0x0104c0},
+ {0x0104c1, 0x0104e9, 0x0104c1, 0x0104c1},
+ {0x0104c2, 0x0104ea, 0x0104c2, 0x0104c2},
+ {0x0104c3, 0x0104eb, 0x0104c3, 0x0104c3},
+ {0x0104c4, 0x0104ec, 0x0104c4, 0x0104c4},
+ {0x0104c5, 0x0104ed, 0x0104c5, 0x0104c5},
+ {0x0104c6, 0x0104ee, 0x0104c6, 0x0104c6},
+ {0x0104c7, 0x0104ef, 0x0104c7, 0x0104c7},
+ {0x0104c8, 0x0104f0, 0x0104c8, 0x0104c8},
+ {0x0104c9, 0x0104f1, 0x0104c9, 0x0104c9},
+ {0x0104ca, 0x0104f2, 0x0104ca, 0x0104ca},
+ {0x0104cb, 0x0104f3, 0x0104cb, 0x0104cb},
+ {0x0104cc, 0x0104f4, 0x0104cc, 0x0104cc},
+ {0x0104cd, 0x0104f5, 0x0104cd, 0x0104cd},
+ {0x0104ce, 0x0104f6, 0x0104ce, 0x0104ce},
+ {0x0104cf, 0x0104f7, 0x0104cf, 0x0104cf},
+ {0x0104d0, 0x0104f8, 0x0104d0, 0x0104d0},
+ {0x0104d1, 0x0104f9, 0x0104d1, 0x0104d1},
+ {0x0104d2, 0x0104fa, 0x0104d2, 0x0104d2},
+ {0x0104d3, 0x0104fb, 0x0104d3, 0x0104d3},
+ {0x0104d8, 0x0104d8, 0x0104b0, 0x0104b0},
+ {0x0104d9, 0x0104d9, 0x0104b1, 0x0104b1},
+ {0x0104da, 0x0104da, 0x0104b2, 0x0104b2},
+ {0x0104db, 0x0104db, 0x0104b3, 0x0104b3},
+ {0x0104dc, 0x0104dc, 0x0104b4, 0x0104b4},
+ {0x0104dd, 0x0104dd, 0x0104b5, 0x0104b5},
+ {0x0104de, 0x0104de, 0x0104b6, 0x0104b6},
+ {0x0104df, 0x0104df, 0x0104b7, 0x0104b7},
+ {0x0104e0, 0x0104e0, 0x0104b8, 0x0104b8},
+ {0x0104e1, 0x0104e1, 0x0104b9, 0x0104b9},
+ {0x0104e2, 0x0104e2, 0x0104ba, 0x0104ba},
+ {0x0104e3, 0x0104e3, 0x0104bb, 0x0104bb},
+ {0x0104e4, 0x0104e4, 0x0104bc, 0x0104bc},
+ {0x0104e5, 0x0104e5, 0x0104bd, 0x0104bd},
+ {0x0104e6, 0x0104e6, 0x0104be, 0x0104be},
+ {0x0104e7, 0x0104e7, 0x0104bf, 0x0104bf},
+ {0x0104e8, 0x0104e8, 0x0104c0, 0x0104c0},
+ {0x0104e9, 0x0104e9, 0x0104c1, 0x0104c1},
+ {0x0104ea, 0x0104ea, 0x0104c2, 0x0104c2},
+ {0x0104eb, 0x0104eb, 0x0104c3, 0x0104c3},
+ {0x0104ec, 0x0104ec, 0x0104c4, 0x0104c4},
+ {0x0104ed, 0x0104ed, 0x0104c5, 0x0104c5},
+ {0x0104ee, 0x0104ee, 0x0104c6, 0x0104c6},
+ {0x0104ef, 0x0104ef, 0x0104c7, 0x0104c7},
+ {0x0104f0, 0x0104f0, 0x0104c8, 0x0104c8},
+ {0x0104f1, 0x0104f1, 0x0104c9, 0x0104c9},
+ {0x0104f2, 0x0104f2, 0x0104ca, 0x0104ca},
+ {0x0104f3, 0x0104f3, 0x0104cb, 0x0104cb},
+ {0x0104f4, 0x0104f4, 0x0104cc, 0x0104cc},
+ {0x0104f5, 0x0104f5, 0x0104cd, 0x0104cd},
+ {0x0104f6, 0x0104f6, 0x0104ce, 0x0104ce},
+ {0x0104f7, 0x0104f7, 0x0104cf, 0x0104cf},
+ {0x0104f8, 0x0104f8, 0x0104d0, 0x0104d0},
+ {0x0104f9, 0x0104f9, 0x0104d1, 0x0104d1},
+ {0x0104fa, 0x0104fa, 0x0104d2, 0x0104d2},
+ {0x0104fb, 0x0104fb, 0x0104d3, 0x0104d3},
+ {0x010570, 0x010597, 0x010570, 0x010570},
+ {0x010571, 0x010598, 0x010571, 0x010571},
+ {0x010572, 0x010599, 0x010572, 0x010572},
+ {0x010573, 0x01059a, 0x010573, 0x010573},
+ {0x010574, 0x01059b, 0x010574, 0x010574},
+ {0x010575, 0x01059c, 0x010575, 0x010575},
+ {0x010576, 0x01059d, 0x010576, 0x010576},
+ {0x010577, 0x01059e, 0x010577, 0x010577},
+ {0x010578, 0x01059f, 0x010578, 0x010578},
+ {0x010579, 0x0105a0, 0x010579, 0x010579},
+ {0x01057a, 0x0105a1, 0x01057a, 0x01057a},
+ {0x01057c, 0x0105a3, 0x01057c, 0x01057c},
+ {0x01057d, 0x0105a4, 0x01057d, 0x01057d},
+ {0x01057e, 0x0105a5, 0x01057e, 0x01057e},
+ {0x01057f, 0x0105a6, 0x01057f, 0x01057f},
+ {0x010580, 0x0105a7, 0x010580, 0x010580},
+ {0x010581, 0x0105a8, 0x010581, 0x010581},
+ {0x010582, 0x0105a9, 0x010582, 0x010582},
+ {0x010583, 0x0105aa, 0x010583, 0x010583},
+ {0x010584, 0x0105ab, 0x010584, 0x010584},
+ {0x010585, 0x0105ac, 0x010585, 0x010585},
+ {0x010586, 0x0105ad, 0x010586, 0x010586},
+ {0x010587, 0x0105ae, 0x010587, 0x010587},
+ {0x010588, 0x0105af, 0x010588, 0x010588},
+ {0x010589, 0x0105b0, 0x010589, 0x010589},
+ {0x01058a, 0x0105b1, 0x01058a, 0x01058a},
+ {0x01058c, 0x0105b3, 0x01058c, 0x01058c},
+ {0x01058d, 0x0105b4, 0x01058d, 0x01058d},
+ {0x01058e, 0x0105b5, 0x01058e, 0x01058e},
+ {0x01058f, 0x0105b6, 0x01058f, 0x01058f},
+ {0x010590, 0x0105b7, 0x010590, 0x010590},
+ {0x010591, 0x0105b8, 0x010591, 0x010591},
+ {0x010592, 0x0105b9, 0x010592, 0x010592},
+ {0x010594, 0x0105bb, 0x010594, 0x010594},
+ {0x010595, 0x0105bc, 0x010595, 0x010595},
+ {0x010597, 0x010597, 0x010570, 0x010570},
+ {0x010598, 0x010598, 0x010571, 0x010571},
+ {0x010599, 0x010599, 0x010572, 0x010572},
+ {0x01059a, 0x01059a, 0x010573, 0x010573},
+ {0x01059b, 0x01059b, 0x010574, 0x010574},
+ {0x01059c, 0x01059c, 0x010575, 0x010575},
+ {0x01059d, 0x01059d, 0x010576, 0x010576},
+ {0x01059e, 0x01059e, 0x010577, 0x010577},
+ {0x01059f, 0x01059f, 0x010578, 0x010578},
+ {0x0105a0, 0x0105a0, 0x010579, 0x010579},
+ {0x0105a1, 0x0105a1, 0x01057a, 0x01057a},
+ {0x0105a3, 0x0105a3, 0x01057c, 0x01057c},
+ {0x0105a4, 0x0105a4, 0x01057d, 0x01057d},
+ {0x0105a5, 0x0105a5, 0x01057e, 0x01057e},
+ {0x0105a6, 0x0105a6, 0x01057f, 0x01057f},
+ {0x0105a7, 0x0105a7, 0x010580, 0x010580},
+ {0x0105a8, 0x0105a8, 0x010581, 0x010581},
+ {0x0105a9, 0x0105a9, 0x010582, 0x010582},
+ {0x0105aa, 0x0105aa, 0x010583, 0x010583},
+ {0x0105ab, 0x0105ab, 0x010584, 0x010584},
+ {0x0105ac, 0x0105ac, 0x010585, 0x010585},
+ {0x0105ad, 0x0105ad, 0x010586, 0x010586},
+ {0x0105ae, 0x0105ae, 0x010587, 0x010587},
+ {0x0105af, 0x0105af, 0x010588, 0x010588},
+ {0x0105b0, 0x0105b0, 0x010589, 0x010589},
+ {0x0105b1, 0x0105b1, 0x01058a, 0x01058a},
+ {0x0105b3, 0x0105b3, 0x01058c, 0x01058c},
+ {0x0105b4, 0x0105b4, 0x01058d, 0x01058d},
+ {0x0105b5, 0x0105b5, 0x01058e, 0x01058e},
+ {0x0105b6, 0x0105b6, 0x01058f, 0x01058f},
+ {0x0105b7, 0x0105b7, 0x010590, 0x010590},
+ {0x0105b8, 0x0105b8, 0x010591, 0x010591},
+ {0x0105b9, 0x0105b9, 0x010592, 0x010592},
+ {0x0105bb, 0x0105bb, 0x010594, 0x010594},
+ {0x0105bc, 0x0105bc, 0x010595, 0x010595},
+ {0x010c80, 0x010cc0, 0x010c80, 0x010c80},
+ {0x010c81, 0x010cc1, 0x010c81, 0x010c81},
+ {0x010c82, 0x010cc2, 0x010c82, 0x010c82},
+ {0x010c83, 0x010cc3, 0x010c83, 0x010c83},
+ {0x010c84, 0x010cc4, 0x010c84, 0x010c84},
+ {0x010c85, 0x010cc5, 0x010c85, 0x010c85},
+ {0x010c86, 0x010cc6, 0x010c86, 0x010c86},
+ {0x010c87, 0x010cc7, 0x010c87, 0x010c87},
+ {0x010c88, 0x010cc8, 0x010c88, 0x010c88},
+ {0x010c89, 0x010cc9, 0x010c89, 0x010c89},
+ {0x010c8a, 0x010cca, 0x010c8a, 0x010c8a},
+ {0x010c8b, 0x010ccb, 0x010c8b, 0x010c8b},
+ {0x010c8c, 0x010ccc, 0x010c8c, 0x010c8c},
+ {0x010c8d, 0x010ccd, 0x010c8d, 0x010c8d},
+ {0x010c8e, 0x010cce, 0x010c8e, 0x010c8e},
+ {0x010c8f, 0x010ccf, 0x010c8f, 0x010c8f},
+ {0x010c90, 0x010cd0, 0x010c90, 0x010c90},
+ {0x010c91, 0x010cd1, 0x010c91, 0x010c91},
+ {0x010c92, 0x010cd2, 0x010c92, 0x010c92},
+ {0x010c93, 0x010cd3, 0x010c93, 0x010c93},
+ {0x010c94, 0x010cd4, 0x010c94, 0x010c94},
+ {0x010c95, 0x010cd5, 0x010c95, 0x010c95},
+ {0x010c96, 0x010cd6, 0x010c96, 0x010c96},
+ {0x010c97, 0x010cd7, 0x010c97, 0x010c97},
+ {0x010c98, 0x010cd8, 0x010c98, 0x010c98},
+ {0x010c99, 0x010cd9, 0x010c99, 0x010c99},
+ {0x010c9a, 0x010cda, 0x010c9a, 0x010c9a},
+ {0x010c9b, 0x010cdb, 0x010c9b, 0x010c9b},
+ {0x010c9c, 0x010cdc, 0x010c9c, 0x010c9c},
+ {0x010c9d, 0x010cdd, 0x010c9d, 0x010c9d},
+ {0x010c9e, 0x010cde, 0x010c9e, 0x010c9e},
+ {0x010c9f, 0x010cdf, 0x010c9f, 0x010c9f},
+ {0x010ca0, 0x010ce0, 0x010ca0, 0x010ca0},
+ {0x010ca1, 0x010ce1, 0x010ca1, 0x010ca1},
+ {0x010ca2, 0x010ce2, 0x010ca2, 0x010ca2},
+ {0x010ca3, 0x010ce3, 0x010ca3, 0x010ca3},
+ {0x010ca4, 0x010ce4, 0x010ca4, 0x010ca4},
+ {0x010ca5, 0x010ce5, 0x010ca5, 0x010ca5},
+ {0x010ca6, 0x010ce6, 0x010ca6, 0x010ca6},
+ {0x010ca7, 0x010ce7, 0x010ca7, 0x010ca7},
+ {0x010ca8, 0x010ce8, 0x010ca8, 0x010ca8},
+ {0x010ca9, 0x010ce9, 0x010ca9, 0x010ca9},
+ {0x010caa, 0x010cea, 0x010caa, 0x010caa},
+ {0x010cab, 0x010ceb, 0x010cab, 0x010cab},
+ {0x010cac, 0x010cec, 0x010cac, 0x010cac},
+ {0x010cad, 0x010ced, 0x010cad, 0x010cad},
+ {0x010cae, 0x010cee, 0x010cae, 0x010cae},
+ {0x010caf, 0x010cef, 0x010caf, 0x010caf},
+ {0x010cb0, 0x010cf0, 0x010cb0, 0x010cb0},
+ {0x010cb1, 0x010cf1, 0x010cb1, 0x010cb1},
+ {0x010cb2, 0x010cf2, 0x010cb2, 0x010cb2},
+ {0x010cc0, 0x010cc0, 0x010c80, 0x010c80},
+ {0x010cc1, 0x010cc1, 0x010c81, 0x010c81},
+ {0x010cc2, 0x010cc2, 0x010c82, 0x010c82},
+ {0x010cc3, 0x010cc3, 0x010c83, 0x010c83},
+ {0x010cc4, 0x010cc4, 0x010c84, 0x010c84},
+ {0x010cc5, 0x010cc5, 0x010c85, 0x010c85},
+ {0x010cc6, 0x010cc6, 0x010c86, 0x010c86},
+ {0x010cc7, 0x010cc7, 0x010c87, 0x010c87},
+ {0x010cc8, 0x010cc8, 0x010c88, 0x010c88},
+ {0x010cc9, 0x010cc9, 0x010c89, 0x010c89},
+ {0x010cca, 0x010cca, 0x010c8a, 0x010c8a},
+ {0x010ccb, 0x010ccb, 0x010c8b, 0x010c8b},
+ {0x010ccc, 0x010ccc, 0x010c8c, 0x010c8c},
+ {0x010ccd, 0x010ccd, 0x010c8d, 0x010c8d},
+ {0x010cce, 0x010cce, 0x010c8e, 0x010c8e},
+ {0x010ccf, 0x010ccf, 0x010c8f, 0x010c8f},
+ {0x010cd0, 0x010cd0, 0x010c90, 0x010c90},
+ {0x010cd1, 0x010cd1, 0x010c91, 0x010c91},
+ {0x010cd2, 0x010cd2, 0x010c92, 0x010c92},
+ {0x010cd3, 0x010cd3, 0x010c93, 0x010c93},
+ {0x010cd4, 0x010cd4, 0x010c94, 0x010c94},
+ {0x010cd5, 0x010cd5, 0x010c95, 0x010c95},
+ {0x010cd6, 0x010cd6, 0x010c96, 0x010c96},
+ {0x010cd7, 0x010cd7, 0x010c97, 0x010c97},
+ {0x010cd8, 0x010cd8, 0x010c98, 0x010c98},
+ {0x010cd9, 0x010cd9, 0x010c99, 0x010c99},
+ {0x010cda, 0x010cda, 0x010c9a, 0x010c9a},
+ {0x010cdb, 0x010cdb, 0x010c9b, 0x010c9b},
+ {0x010cdc, 0x010cdc, 0x010c9c, 0x010c9c},
+ {0x010cdd, 0x010cdd, 0x010c9d, 0x010c9d},
+ {0x010cde, 0x010cde, 0x010c9e, 0x010c9e},
+ {0x010cdf, 0x010cdf, 0x010c9f, 0x010c9f},
+ {0x010ce0, 0x010ce0, 0x010ca0, 0x010ca0},
+ {0x010ce1, 0x010ce1, 0x010ca1, 0x010ca1},
+ {0x010ce2, 0x010ce2, 0x010ca2, 0x010ca2},
+ {0x010ce3, 0x010ce3, 0x010ca3, 0x010ca3},
+ {0x010ce4, 0x010ce4, 0x010ca4, 0x010ca4},
+ {0x010ce5, 0x010ce5, 0x010ca5, 0x010ca5},
+ {0x010ce6, 0x010ce6, 0x010ca6, 0x010ca6},
+ {0x010ce7, 0x010ce7, 0x010ca7, 0x010ca7},
+ {0x010ce8, 0x010ce8, 0x010ca8, 0x010ca8},
+ {0x010ce9, 0x010ce9, 0x010ca9, 0x010ca9},
+ {0x010cea, 0x010cea, 0x010caa, 0x010caa},
+ {0x010ceb, 0x010ceb, 0x010cab, 0x010cab},
+ {0x010cec, 0x010cec, 0x010cac, 0x010cac},
+ {0x010ced, 0x010ced, 0x010cad, 0x010cad},
+ {0x010cee, 0x010cee, 0x010cae, 0x010cae},
+ {0x010cef, 0x010cef, 0x010caf, 0x010caf},
+ {0x010cf0, 0x010cf0, 0x010cb0, 0x010cb0},
+ {0x010cf1, 0x010cf1, 0x010cb1, 0x010cb1},
+ {0x010cf2, 0x010cf2, 0x010cb2, 0x010cb2},
+ {0x0118a0, 0x0118c0, 0x0118a0, 0x0118a0},
+ {0x0118a1, 0x0118c1, 0x0118a1, 0x0118a1},
+ {0x0118a2, 0x0118c2, 0x0118a2, 0x0118a2},
+ {0x0118a3, 0x0118c3, 0x0118a3, 0x0118a3},
+ {0x0118a4, 0x0118c4, 0x0118a4, 0x0118a4},
+ {0x0118a5, 0x0118c5, 0x0118a5, 0x0118a5},
+ {0x0118a6, 0x0118c6, 0x0118a6, 0x0118a6},
+ {0x0118a7, 0x0118c7, 0x0118a7, 0x0118a7},
+ {0x0118a8, 0x0118c8, 0x0118a8, 0x0118a8},
+ {0x0118a9, 0x0118c9, 0x0118a9, 0x0118a9},
+ {0x0118aa, 0x0118ca, 0x0118aa, 0x0118aa},
+ {0x0118ab, 0x0118cb, 0x0118ab, 0x0118ab},
+ {0x0118ac, 0x0118cc, 0x0118ac, 0x0118ac},
+ {0x0118ad, 0x0118cd, 0x0118ad, 0x0118ad},
+ {0x0118ae, 0x0118ce, 0x0118ae, 0x0118ae},
+ {0x0118af, 0x0118cf, 0x0118af, 0x0118af},
+ {0x0118b0, 0x0118d0, 0x0118b0, 0x0118b0},
+ {0x0118b1, 0x0118d1, 0x0118b1, 0x0118b1},
+ {0x0118b2, 0x0118d2, 0x0118b2, 0x0118b2},
+ {0x0118b3, 0x0118d3, 0x0118b3, 0x0118b3},
+ {0x0118b4, 0x0118d4, 0x0118b4, 0x0118b4},
+ {0x0118b5, 0x0118d5, 0x0118b5, 0x0118b5},
+ {0x0118b6, 0x0118d6, 0x0118b6, 0x0118b6},
+ {0x0118b7, 0x0118d7, 0x0118b7, 0x0118b7},
+ {0x0118b8, 0x0118d8, 0x0118b8, 0x0118b8},
+ {0x0118b9, 0x0118d9, 0x0118b9, 0x0118b9},
+ {0x0118ba, 0x0118da, 0x0118ba, 0x0118ba},
+ {0x0118bb, 0x0118db, 0x0118bb, 0x0118bb},
+ {0x0118bc, 0x0118dc, 0x0118bc, 0x0118bc},
+ {0x0118bd, 0x0118dd, 0x0118bd, 0x0118bd},
+ {0x0118be, 0x0118de, 0x0118be, 0x0118be},
+ {0x0118bf, 0x0118df, 0x0118bf, 0x0118bf},
+ {0x0118c0, 0x0118c0, 0x0118a0, 0x0118a0},
+ {0x0118c1, 0x0118c1, 0x0118a1, 0x0118a1},
+ {0x0118c2, 0x0118c2, 0x0118a2, 0x0118a2},
+ {0x0118c3, 0x0118c3, 0x0118a3, 0x0118a3},
+ {0x0118c4, 0x0118c4, 0x0118a4, 0x0118a4},
+ {0x0118c5, 0x0118c5, 0x0118a5, 0x0118a5},
+ {0x0118c6, 0x0118c6, 0x0118a6, 0x0118a6},
+ {0x0118c7, 0x0118c7, 0x0118a7, 0x0118a7},
+ {0x0118c8, 0x0118c8, 0x0118a8, 0x0118a8},
+ {0x0118c9, 0x0118c9, 0x0118a9, 0x0118a9},
+ {0x0118ca, 0x0118ca, 0x0118aa, 0x0118aa},
+ {0x0118cb, 0x0118cb, 0x0118ab, 0x0118ab},
+ {0x0118cc, 0x0118cc, 0x0118ac, 0x0118ac},
+ {0x0118cd, 0x0118cd, 0x0118ad, 0x0118ad},
+ {0x0118ce, 0x0118ce, 0x0118ae, 0x0118ae},
+ {0x0118cf, 0x0118cf, 0x0118af, 0x0118af},
+ {0x0118d0, 0x0118d0, 0x0118b0, 0x0118b0},
+ {0x0118d1, 0x0118d1, 0x0118b1, 0x0118b1},
+ {0x0118d2, 0x0118d2, 0x0118b2, 0x0118b2},
+ {0x0118d3, 0x0118d3, 0x0118b3, 0x0118b3},
+ {0x0118d4, 0x0118d4, 0x0118b4, 0x0118b4},
+ {0x0118d5, 0x0118d5, 0x0118b5, 0x0118b5},
+ {0x0118d6, 0x0118d6, 0x0118b6, 0x0118b6},
+ {0x0118d7, 0x0118d7, 0x0118b7, 0x0118b7},
+ {0x0118d8, 0x0118d8, 0x0118b8, 0x0118b8},
+ {0x0118d9, 0x0118d9, 0x0118b9, 0x0118b9},
+ {0x0118da, 0x0118da, 0x0118ba, 0x0118ba},
+ {0x0118db, 0x0118db, 0x0118bb, 0x0118bb},
+ {0x0118dc, 0x0118dc, 0x0118bc, 0x0118bc},
+ {0x0118dd, 0x0118dd, 0x0118bd, 0x0118bd},
+ {0x0118de, 0x0118de, 0x0118be, 0x0118be},
+ {0x0118df, 0x0118df, 0x0118bf, 0x0118bf},
+ {0x016e40, 0x016e60, 0x016e40, 0x016e40},
+ {0x016e41, 0x016e61, 0x016e41, 0x016e41},
+ {0x016e42, 0x016e62, 0x016e42, 0x016e42},
+ {0x016e43, 0x016e63, 0x016e43, 0x016e43},
+ {0x016e44, 0x016e64, 0x016e44, 0x016e44},
+ {0x016e45, 0x016e65, 0x016e45, 0x016e45},
+ {0x016e46, 0x016e66, 0x016e46, 0x016e46},
+ {0x016e47, 0x016e67, 0x016e47, 0x016e47},
+ {0x016e48, 0x016e68, 0x016e48, 0x016e48},
+ {0x016e49, 0x016e69, 0x016e49, 0x016e49},
+ {0x016e4a, 0x016e6a, 0x016e4a, 0x016e4a},
+ {0x016e4b, 0x016e6b, 0x016e4b, 0x016e4b},
+ {0x016e4c, 0x016e6c, 0x016e4c, 0x016e4c},
+ {0x016e4d, 0x016e6d, 0x016e4d, 0x016e4d},
+ {0x016e4e, 0x016e6e, 0x016e4e, 0x016e4e},
+ {0x016e4f, 0x016e6f, 0x016e4f, 0x016e4f},
+ {0x016e50, 0x016e70, 0x016e50, 0x016e50},
+ {0x016e51, 0x016e71, 0x016e51, 0x016e51},
+ {0x016e52, 0x016e72, 0x016e52, 0x016e52},
+ {0x016e53, 0x016e73, 0x016e53, 0x016e53},
+ {0x016e54, 0x016e74, 0x016e54, 0x016e54},
+ {0x016e55, 0x016e75, 0x016e55, 0x016e55},
+ {0x016e56, 0x016e76, 0x016e56, 0x016e56},
+ {0x016e57, 0x016e77, 0x016e57, 0x016e57},
+ {0x016e58, 0x016e78, 0x016e58, 0x016e58},
+ {0x016e59, 0x016e79, 0x016e59, 0x016e59},
+ {0x016e5a, 0x016e7a, 0x016e5a, 0x016e5a},
+ {0x016e5b, 0x016e7b, 0x016e5b, 0x016e5b},
+ {0x016e5c, 0x016e7c, 0x016e5c, 0x016e5c},
+ {0x016e5d, 0x016e7d, 0x016e5d, 0x016e5d},
+ {0x016e5e, 0x016e7e, 0x016e5e, 0x016e5e},
+ {0x016e5f, 0x016e7f, 0x016e5f, 0x016e5f},
+ {0x016e60, 0x016e60, 0x016e40, 0x016e40},
+ {0x016e61, 0x016e61, 0x016e41, 0x016e41},
+ {0x016e62, 0x016e62, 0x016e42, 0x016e42},
+ {0x016e63, 0x016e63, 0x016e43, 0x016e43},
+ {0x016e64, 0x016e64, 0x016e44, 0x016e44},
+ {0x016e65, 0x016e65, 0x016e45, 0x016e45},
+ {0x016e66, 0x016e66, 0x016e46, 0x016e46},
+ {0x016e67, 0x016e67, 0x016e47, 0x016e47},
+ {0x016e68, 0x016e68, 0x016e48, 0x016e48},
+ {0x016e69, 0x016e69, 0x016e49, 0x016e49},
+ {0x016e6a, 0x016e6a, 0x016e4a, 0x016e4a},
+ {0x016e6b, 0x016e6b, 0x016e4b, 0x016e4b},
+ {0x016e6c, 0x016e6c, 0x016e4c, 0x016e4c},
+ {0x016e6d, 0x016e6d, 0x016e4d, 0x016e4d},
+ {0x016e6e, 0x016e6e, 0x016e4e, 0x016e4e},
+ {0x016e6f, 0x016e6f, 0x016e4f, 0x016e4f},
+ {0x016e70, 0x016e70, 0x016e50, 0x016e50},
+ {0x016e71, 0x016e71, 0x016e51, 0x016e51},
+ {0x016e72, 0x016e72, 0x016e52, 0x016e52},
+ {0x016e73, 0x016e73, 0x016e53, 0x016e53},
+ {0x016e74, 0x016e74, 0x016e54, 0x016e54},
+ {0x016e75, 0x016e75, 0x016e55, 0x016e55},
+ {0x016e76, 0x016e76, 0x016e56, 0x016e56},
+ {0x016e77, 0x016e77, 0x016e57, 0x016e57},
+ {0x016e78, 0x016e78, 0x016e58, 0x016e58},
+ {0x016e79, 0x016e79, 0x016e59, 0x016e59},
+ {0x016e7a, 0x016e7a, 0x016e5a, 0x016e5a},
+ {0x016e7b, 0x016e7b, 0x016e5b, 0x016e5b},
+ {0x016e7c, 0x016e7c, 0x016e5c, 0x016e5c},
+ {0x016e7d, 0x016e7d, 0x016e5d, 0x016e5d},
+ {0x016e7e, 0x016e7e, 0x016e5e, 0x016e5e},
+ {0x016e7f, 0x016e7f, 0x016e5f, 0x016e5f},
+ {0x01e900, 0x01e922, 0x01e900, 0x01e900},
+ {0x01e901, 0x01e923, 0x01e901, 0x01e901},
+ {0x01e902, 0x01e924, 0x01e902, 0x01e902},
+ {0x01e903, 0x01e925, 0x01e903, 0x01e903},
+ {0x01e904, 0x01e926, 0x01e904, 0x01e904},
+ {0x01e905, 0x01e927, 0x01e905, 0x01e905},
+ {0x01e906, 0x01e928, 0x01e906, 0x01e906},
+ {0x01e907, 0x01e929, 0x01e907, 0x01e907},
+ {0x01e908, 0x01e92a, 0x01e908, 0x01e908},
+ {0x01e909, 0x01e92b, 0x01e909, 0x01e909},
+ {0x01e90a, 0x01e92c, 0x01e90a, 0x01e90a},
+ {0x01e90b, 0x01e92d, 0x01e90b, 0x01e90b},
+ {0x01e90c, 0x01e92e, 0x01e90c, 0x01e90c},
+ {0x01e90d, 0x01e92f, 0x01e90d, 0x01e90d},
+ {0x01e90e, 0x01e930, 0x01e90e, 0x01e90e},
+ {0x01e90f, 0x01e931, 0x01e90f, 0x01e90f},
+ {0x01e910, 0x01e932, 0x01e910, 0x01e910},
+ {0x01e911, 0x01e933, 0x01e911, 0x01e911},
+ {0x01e912, 0x01e934, 0x01e912, 0x01e912},
+ {0x01e913, 0x01e935, 0x01e913, 0x01e913},
+ {0x01e914, 0x01e936, 0x01e914, 0x01e914},
+ {0x01e915, 0x01e937, 0x01e915, 0x01e915},
+ {0x01e916, 0x01e938, 0x01e916, 0x01e916},
+ {0x01e917, 0x01e939, 0x01e917, 0x01e917},
+ {0x01e918, 0x01e93a, 0x01e918, 0x01e918},
+ {0x01e919, 0x01e93b, 0x01e919, 0x01e919},
+ {0x01e91a, 0x01e93c, 0x01e91a, 0x01e91a},
+ {0x01e91b, 0x01e93d, 0x01e91b, 0x01e91b},
+ {0x01e91c, 0x01e93e, 0x01e91c, 0x01e91c},
+ {0x01e91d, 0x01e93f, 0x01e91d, 0x01e91d},
+ {0x01e91e, 0x01e940, 0x01e91e, 0x01e91e},
+ {0x01e91f, 0x01e941, 0x01e91f, 0x01e91f},
+ {0x01e920, 0x01e942, 0x01e920, 0x01e920},
+ {0x01e921, 0x01e943, 0x01e921, 0x01e921},
+ {0x01e922, 0x01e922, 0x01e900, 0x01e900},
+ {0x01e923, 0x01e923, 0x01e901, 0x01e901},
+ {0x01e924, 0x01e924, 0x01e902, 0x01e902},
+ {0x01e925, 0x01e925, 0x01e903, 0x01e903},
+ {0x01e926, 0x01e926, 0x01e904, 0x01e904},
+ {0x01e927, 0x01e927, 0x01e905, 0x01e905},
+ {0x01e928, 0x01e928, 0x01e906, 0x01e906},
+ {0x01e929, 0x01e929, 0x01e907, 0x01e907},
+ {0x01e92a, 0x01e92a, 0x01e908, 0x01e908},
+ {0x01e92b, 0x01e92b, 0x01e909, 0x01e909},
+ {0x01e92c, 0x01e92c, 0x01e90a, 0x01e90a},
+ {0x01e92d, 0x01e92d, 0x01e90b, 0x01e90b},
+ {0x01e92e, 0x01e92e, 0x01e90c, 0x01e90c},
+ {0x01e92f, 0x01e92f, 0x01e90d, 0x01e90d},
+ {0x01e930, 0x01e930, 0x01e90e, 0x01e90e},
+ {0x01e931, 0x01e931, 0x01e90f, 0x01e90f},
+ {0x01e932, 0x01e932, 0x01e910, 0x01e910},
+ {0x01e933, 0x01e933, 0x01e911, 0x01e911},
+ {0x01e934, 0x01e934, 0x01e912, 0x01e912},
+ {0x01e935, 0x01e935, 0x01e913, 0x01e913},
+ {0x01e936, 0x01e936, 0x01e914, 0x01e914},
+ {0x01e937, 0x01e937, 0x01e915, 0x01e915},
+ {0x01e938, 0x01e938, 0x01e916, 0x01e916},
+ {0x01e939, 0x01e939, 0x01e917, 0x01e917},
+ {0x01e93a, 0x01e93a, 0x01e918, 0x01e918},
+ {0x01e93b, 0x01e93b, 0x01e919, 0x01e919},
+ {0x01e93c, 0x01e93c, 0x01e91a, 0x01e91a},
+ {0x01e93d, 0x01e93d, 0x01e91b, 0x01e91b},
+ {0x01e93e, 0x01e93e, 0x01e91c, 0x01e91c},
+ {0x01e93f, 0x01e93f, 0x01e91d, 0x01e91d},
+ {0x01e940, 0x01e940, 0x01e91e, 0x01e91e},
+ {0x01e941, 0x01e941, 0x01e91f, 0x01e91f},
+ {0x01e942, 0x01e942, 0x01e920, 0x01e920},
+ {0x01e943, 0x01e943, 0x01e921, 0x01e921}
+};
--
2.34.1
[text/x-patch] v15-0004-Catalog-changes-preparing-for-builtin-collation-.patch (46.2K, ../../[email protected]/5-v15-0004-Catalog-changes-preparing-for-builtin-collation-.patch)
download | inline diff:
From 15feab3483f0f31a65bf3621f553af8ae47de55d Mon Sep 17 00:00:00 2001
From: Jeff Davis <[email protected]>
Date: Tue, 26 Dec 2023 13:32:48 -0800
Subject: [PATCH v15 4/5] Catalog changes preparing for builtin collation
provider.
daticulocale -> datlocale, colliculocale -> colllocale.
---
src/backend/catalog/pg_collation.c | 10 +--
src/backend/commands/collationcmds.c | 34 +++----
src/backend/commands/dbcommands.c | 68 +++++++-------
src/backend/utils/adt/pg_locale.c | 4 +-
src/backend/utils/init/postinit.c | 2 +-
src/bin/initdb/initdb.c | 41 ++++-----
src/bin/initdb/t/001_initdb.pl | 4 +-
src/bin/pg_dump/pg_dump.c | 90 +++++++++++--------
src/bin/pg_upgrade/info.c | 31 ++++---
src/bin/pg_upgrade/pg_upgrade.c | 34 ++++---
src/bin/pg_upgrade/pg_upgrade.h | 2 +-
src/bin/pg_upgrade/t/002_pg_upgrade.pl | 13 ++-
src/bin/psql/describe.c | 20 +++--
src/include/catalog/pg_collation.dat | 2 +-
src/include/catalog/pg_collation.h | 4 +-
src/include/catalog/pg_database.dat | 2 +-
src/include/catalog/pg_database.h | 2 +-
.../regress/expected/collate.icu.utf8.out | 4 +-
src/test/regress/expected/psql.out | 18 ++--
src/test/regress/sql/collate.icu.utf8.sql | 4 +-
20 files changed, 220 insertions(+), 169 deletions(-)
diff --git a/src/backend/catalog/pg_collation.c b/src/backend/catalog/pg_collation.c
index fd022e6fc2..b615d2fc7a 100644
--- a/src/backend/catalog/pg_collation.c
+++ b/src/backend/catalog/pg_collation.c
@@ -49,7 +49,7 @@ CollationCreate(const char *collname, Oid collnamespace,
bool collisdeterministic,
int32 collencoding,
const char *collcollate, const char *collctype,
- const char *colliculocale,
+ const char *colllocale,
const char *collicurules,
const char *collversion,
bool if_not_exists,
@@ -68,7 +68,7 @@ CollationCreate(const char *collname, Oid collnamespace,
Assert(collname);
Assert(collnamespace);
Assert(collowner);
- Assert((collcollate && collctype) || colliculocale);
+ Assert((collcollate && collctype) || colllocale);
/*
* Make sure there is no existing collation of same name & encoding.
@@ -191,10 +191,10 @@ CollationCreate(const char *collname, Oid collnamespace,
values[Anum_pg_collation_collctype - 1] = CStringGetTextDatum(collctype);
else
nulls[Anum_pg_collation_collctype - 1] = true;
- if (colliculocale)
- values[Anum_pg_collation_colliculocale - 1] = CStringGetTextDatum(colliculocale);
+ if (colllocale)
+ values[Anum_pg_collation_colllocale - 1] = CStringGetTextDatum(colllocale);
else
- nulls[Anum_pg_collation_colliculocale - 1] = true;
+ nulls[Anum_pg_collation_colllocale - 1] = true;
if (collicurules)
values[Anum_pg_collation_collicurules - 1] = CStringGetTextDatum(collicurules);
else
diff --git a/src/backend/commands/collationcmds.c b/src/backend/commands/collationcmds.c
index a4532bf81f..19e61b3e5b 100644
--- a/src/backend/commands/collationcmds.c
+++ b/src/backend/commands/collationcmds.c
@@ -68,7 +68,7 @@ DefineCollation(ParseState *pstate, List *names, List *parameters, bool if_not_e
DefElem *versionEl = NULL;
char *collcollate;
char *collctype;
- char *colliculocale;
+ char *colllocale;
char *collicurules;
bool collisdeterministic;
int collencoding;
@@ -159,11 +159,11 @@ DefineCollation(ParseState *pstate, List *names, List *parameters, bool if_not_e
else
collctype = NULL;
- datum = SysCacheGetAttr(COLLOID, tp, Anum_pg_collation_colliculocale, &isnull);
+ datum = SysCacheGetAttr(COLLOID, tp, Anum_pg_collation_colllocale, &isnull);
if (!isnull)
- colliculocale = TextDatumGetCString(datum);
+ colllocale = TextDatumGetCString(datum);
else
- colliculocale = NULL;
+ colllocale = NULL;
/*
* When the ICU locale comes from an existing collation, do not
@@ -196,7 +196,7 @@ DefineCollation(ParseState *pstate, List *names, List *parameters, bool if_not_e
collcollate = NULL;
collctype = NULL;
- colliculocale = NULL;
+ colllocale = NULL;
collicurules = NULL;
if (providerEl)
@@ -236,7 +236,7 @@ DefineCollation(ParseState *pstate, List *names, List *parameters, bool if_not_e
collctype = defGetString(localeEl);
}
else
- colliculocale = defGetString(localeEl);
+ colllocale = defGetString(localeEl);
}
if (lccollateEl)
@@ -261,7 +261,7 @@ DefineCollation(ParseState *pstate, List *names, List *parameters, bool if_not_e
}
else if (collprovider == COLLPROVIDER_ICU)
{
- if (!colliculocale)
+ if (!colllocale)
ereport(ERROR,
(errcode(ERRCODE_INVALID_OBJECT_DEFINITION),
errmsg("parameter \"%s\" must be specified",
@@ -273,20 +273,20 @@ DefineCollation(ParseState *pstate, List *names, List *parameters, bool if_not_e
*/
if (!IsBinaryUpgrade)
{
- char *langtag = icu_language_tag(colliculocale,
+ char *langtag = icu_language_tag(colllocale,
icu_validation_level);
- if (langtag && strcmp(colliculocale, langtag) != 0)
+ if (langtag && strcmp(colllocale, langtag) != 0)
{
ereport(NOTICE,
(errmsg("using standard form \"%s\" for ICU locale \"%s\"",
- langtag, colliculocale)));
+ langtag, colllocale)));
- colliculocale = langtag;
+ colllocale = langtag;
}
}
- icu_validate_locale(colliculocale);
+ icu_validate_locale(colllocale);
}
/*
@@ -334,7 +334,7 @@ DefineCollation(ParseState *pstate, List *names, List *parameters, bool if_not_e
}
if (!collversion)
- collversion = get_collation_actual_version(collprovider, collprovider == COLLPROVIDER_ICU ? colliculocale : collcollate);
+ collversion = get_collation_actual_version(collprovider, collprovider == COLLPROVIDER_ICU ? colllocale : collcollate);
newoid = CollationCreate(collName,
collNamespace,
@@ -344,7 +344,7 @@ DefineCollation(ParseState *pstate, List *names, List *parameters, bool if_not_e
collencoding,
collcollate,
collctype,
- colliculocale,
+ colllocale,
collicurules,
collversion,
if_not_exists,
@@ -435,7 +435,7 @@ AlterCollation(AlterCollationStmt *stmt)
datum = SysCacheGetAttr(COLLOID, tup, Anum_pg_collation_collversion, &isnull);
oldversion = isnull ? NULL : TextDatumGetCString(datum);
- datum = SysCacheGetAttrNotNull(COLLOID, tup, collForm->collprovider == COLLPROVIDER_ICU ? Anum_pg_collation_colliculocale : Anum_pg_collation_collcollate);
+ datum = SysCacheGetAttrNotNull(COLLOID, tup, collForm->collprovider == COLLPROVIDER_ICU ? Anum_pg_collation_colllocale : Anum_pg_collation_collcollate);
newversion = get_collation_actual_version(collForm->collprovider, TextDatumGetCString(datum));
/* cannot change from NULL to non-NULL or vice versa */
@@ -502,7 +502,7 @@ pg_collation_actual_version(PG_FUNCTION_ARGS)
datum = SysCacheGetAttrNotNull(DATABASEOID, dbtup,
provider == COLLPROVIDER_ICU ?
- Anum_pg_database_daticulocale : Anum_pg_database_datcollate);
+ Anum_pg_database_datlocale : Anum_pg_database_datcollate);
locale = TextDatumGetCString(datum);
@@ -523,7 +523,7 @@ pg_collation_actual_version(PG_FUNCTION_ARGS)
Assert(provider != COLLPROVIDER_DEFAULT);
datum = SysCacheGetAttrNotNull(COLLOID, colltp,
provider == COLLPROVIDER_ICU ?
- Anum_pg_collation_colliculocale : Anum_pg_collation_collcollate);
+ Anum_pg_collation_colllocale : Anum_pg_collation_collcollate);
locale = TextDatumGetCString(datum);
diff --git a/src/backend/commands/dbcommands.c b/src/backend/commands/dbcommands.c
index ae38f83024..c8732be067 100644
--- a/src/backend/commands/dbcommands.c
+++ b/src/backend/commands/dbcommands.c
@@ -118,7 +118,7 @@ static bool get_db_info(const char *name, LOCKMODE lockmode,
Oid *dbIdP, Oid *ownerIdP,
int *encodingP, bool *dbIsTemplateP, bool *dbAllowConnP, bool *dbHasLoginEvtP,
TransactionId *dbFrozenXidP, MultiXactId *dbMinMultiP,
- Oid *dbTablespace, char **dbCollate, char **dbCtype, char **dbIculocale,
+ Oid *dbTablespace, char **dbCollate, char **dbCtype, char **dbLocale,
char **dbIcurules,
char *dbLocProvider,
char **dbCollversion);
@@ -675,7 +675,7 @@ createdb(ParseState *pstate, const CreatedbStmt *stmt)
int src_encoding = -1;
char *src_collate = NULL;
char *src_ctype = NULL;
- char *src_iculocale = NULL;
+ char *src_locale = NULL;
char *src_icurules = NULL;
char src_locprovider = '\0';
char *src_collversion = NULL;
@@ -713,7 +713,7 @@ createdb(ParseState *pstate, const CreatedbStmt *stmt)
const char *dbtemplate = NULL;
char *dbcollate = NULL;
char *dbctype = NULL;
- char *dbiculocale = NULL;
+ char *dblocale = NULL;
char *dbicurules = NULL;
char dblocprovider = '\0';
char *canonname;
@@ -903,7 +903,7 @@ createdb(ParseState *pstate, const CreatedbStmt *stmt)
if (dctype && dctype->arg)
dbctype = defGetString(dctype);
if (diculocale && diculocale->arg)
- dbiculocale = defGetString(diculocale);
+ dblocale = defGetString(diculocale);
if (dicurules && dicurules->arg)
dbicurules = defGetString(dicurules);
if (dlocprovider && dlocprovider->arg)
@@ -971,7 +971,7 @@ createdb(ParseState *pstate, const CreatedbStmt *stmt)
&src_dboid, &src_owner, &src_encoding,
&src_istemplate, &src_allowconn, &src_hasloginevt,
&src_frozenxid, &src_minmxid, &src_deftablespace,
- &src_collate, &src_ctype, &src_iculocale, &src_icurules, &src_locprovider,
+ &src_collate, &src_ctype, &src_locale, &src_icurules, &src_locprovider,
&src_collversion))
ereport(ERROR,
(errcode(ERRCODE_UNDEFINED_DATABASE),
@@ -1027,12 +1027,12 @@ createdb(ParseState *pstate, const CreatedbStmt *stmt)
dbctype = src_ctype;
if (dblocprovider == '\0')
dblocprovider = src_locprovider;
- if (dbiculocale == NULL && dblocprovider == COLLPROVIDER_ICU)
+ if (dblocale == NULL && dblocprovider == COLLPROVIDER_ICU)
{
if (dlocale && dlocale->arg)
- dbiculocale = defGetString(dlocale);
+ dblocale = defGetString(dlocale);
else
- dbiculocale = src_iculocale;
+ dblocale = src_locale;
}
if (dbicurules == NULL && dblocprovider == COLLPROVIDER_ICU)
dbicurules = src_icurules;
@@ -1071,7 +1071,7 @@ createdb(ParseState *pstate, const CreatedbStmt *stmt)
* This would happen if template0 uses the libc provider but the new
* database uses icu.
*/
- if (!dbiculocale)
+ if (!dblocale)
ereport(ERROR,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("LOCALE or ICU_LOCALE must be specified")));
@@ -1081,26 +1081,26 @@ createdb(ParseState *pstate, const CreatedbStmt *stmt)
* database, preserve locale string. Otherwise, canonicalize to a
* language tag.
*/
- if (!IsBinaryUpgrade && dbiculocale != src_iculocale)
+ if (!IsBinaryUpgrade && dblocale != src_locale)
{
- char *langtag = icu_language_tag(dbiculocale,
+ char *langtag = icu_language_tag(dblocale,
icu_validation_level);
- if (langtag && strcmp(dbiculocale, langtag) != 0)
+ if (langtag && strcmp(dblocale, langtag) != 0)
{
ereport(NOTICE,
(errmsg("using standard form \"%s\" for ICU locale \"%s\"",
- langtag, dbiculocale)));
+ langtag, dblocale)));
- dbiculocale = langtag;
+ dblocale = langtag;
}
}
- icu_validate_locale(dbiculocale);
+ icu_validate_locale(dblocale);
}
else
{
- if (dbiculocale)
+ if (dblocale)
ereport(ERROR,
(errcode(ERRCODE_INVALID_OBJECT_DEFINITION),
errmsg("ICU locale cannot be specified unless locale provider is ICU")));
@@ -1157,13 +1157,13 @@ createdb(ParseState *pstate, const CreatedbStmt *stmt)
char *val1;
char *val2;
- Assert(dbiculocale);
- Assert(src_iculocale);
- if (strcmp(dbiculocale, src_iculocale) != 0)
+ Assert(dblocale);
+ Assert(src_locale);
+ if (strcmp(dblocale, src_locale) != 0)
ereport(ERROR,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("new ICU locale (%s) is incompatible with the ICU locale of the template database (%s)",
- dbiculocale, src_iculocale),
+ dblocale, src_locale),
errhint("Use the same ICU locale as in the template database, or use template0 as template.")));
val1 = dbicurules;
@@ -1197,7 +1197,7 @@ createdb(ParseState *pstate, const CreatedbStmt *stmt)
{
char *actual_versionstr;
- actual_versionstr = get_collation_actual_version(dblocprovider, dblocprovider == COLLPROVIDER_ICU ? dbiculocale : dbcollate);
+ actual_versionstr = get_collation_actual_version(dblocprovider, dblocprovider == COLLPROVIDER_ICU ? dblocale : dbcollate);
if (!actual_versionstr)
ereport(ERROR,
(errmsg("template database \"%s\" has a collation version, but no actual collation version could be determined",
@@ -1225,7 +1225,7 @@ createdb(ParseState *pstate, const CreatedbStmt *stmt)
* collation version, which is normally only the case for template0.
*/
if (dbcollversion == NULL)
- dbcollversion = get_collation_actual_version(dblocprovider, dblocprovider == COLLPROVIDER_ICU ? dbiculocale : dbcollate);
+ dbcollversion = get_collation_actual_version(dblocprovider, dblocprovider == COLLPROVIDER_ICU ? dblocale : dbcollate);
/* Resolve default tablespace for new database */
if (dtablespacename && dtablespacename->arg)
@@ -1364,8 +1364,8 @@ createdb(ParseState *pstate, const CreatedbStmt *stmt)
* block on the unique index, and fail after we commit).
*/
- Assert((dblocprovider == COLLPROVIDER_ICU && dbiculocale) ||
- (dblocprovider != COLLPROVIDER_ICU && !dbiculocale));
+ Assert((dblocprovider == COLLPROVIDER_ICU && dblocale) ||
+ (dblocprovider != COLLPROVIDER_ICU && !dblocale));
/* Form tuple */
new_record[Anum_pg_database_oid - 1] = ObjectIdGetDatum(dboid);
@@ -1383,10 +1383,10 @@ createdb(ParseState *pstate, const CreatedbStmt *stmt)
new_record[Anum_pg_database_dattablespace - 1] = ObjectIdGetDatum(dst_deftablespace);
new_record[Anum_pg_database_datcollate - 1] = CStringGetTextDatum(dbcollate);
new_record[Anum_pg_database_datctype - 1] = CStringGetTextDatum(dbctype);
- if (dbiculocale)
- new_record[Anum_pg_database_daticulocale - 1] = CStringGetTextDatum(dbiculocale);
+ if (dblocale)
+ new_record[Anum_pg_database_datlocale - 1] = CStringGetTextDatum(dblocale);
else
- new_record_nulls[Anum_pg_database_daticulocale - 1] = true;
+ new_record_nulls[Anum_pg_database_datlocale - 1] = true;
if (dbicurules)
new_record[Anum_pg_database_daticurules - 1] = CStringGetTextDatum(dbicurules);
else
@@ -2472,7 +2472,7 @@ AlterDatabaseRefreshColl(AlterDatabaseRefreshCollStmt *stmt)
datum = heap_getattr(tuple, Anum_pg_database_datcollversion, RelationGetDescr(rel), &isnull);
oldversion = isnull ? NULL : TextDatumGetCString(datum);
- datum = heap_getattr(tuple, datForm->datlocprovider == COLLPROVIDER_ICU ? Anum_pg_database_daticulocale : Anum_pg_database_datcollate, RelationGetDescr(rel), &isnull);
+ datum = heap_getattr(tuple, datForm->datlocprovider == COLLPROVIDER_ICU ? Anum_pg_database_datlocale : Anum_pg_database_datcollate, RelationGetDescr(rel), &isnull);
if (isnull)
elog(ERROR, "unexpected null in pg_database");
newversion = get_collation_actual_version(datForm->datlocprovider, TextDatumGetCString(datum));
@@ -2670,7 +2670,7 @@ pg_database_collation_actual_version(PG_FUNCTION_ARGS)
datlocprovider = ((Form_pg_database) GETSTRUCT(tp))->datlocprovider;
- datum = SysCacheGetAttrNotNull(DATABASEOID, tp, datlocprovider == COLLPROVIDER_ICU ? Anum_pg_database_daticulocale : Anum_pg_database_datcollate);
+ datum = SysCacheGetAttrNotNull(DATABASEOID, tp, datlocprovider == COLLPROVIDER_ICU ? Anum_pg_database_datlocale : Anum_pg_database_datcollate);
version = get_collation_actual_version(datlocprovider, TextDatumGetCString(datum));
ReleaseSysCache(tp);
@@ -2697,7 +2697,7 @@ get_db_info(const char *name, LOCKMODE lockmode,
Oid *dbIdP, Oid *ownerIdP,
int *encodingP, bool *dbIsTemplateP, bool *dbAllowConnP, bool *dbHasLoginEvtP,
TransactionId *dbFrozenXidP, MultiXactId *dbMinMultiP,
- Oid *dbTablespace, char **dbCollate, char **dbCtype, char **dbIculocale,
+ Oid *dbTablespace, char **dbCollate, char **dbCtype, char **dbLocale,
char **dbIcurules,
char *dbLocProvider,
char **dbCollversion)
@@ -2808,13 +2808,13 @@ get_db_info(const char *name, LOCKMODE lockmode,
datum = SysCacheGetAttrNotNull(DATABASEOID, tuple, Anum_pg_database_datctype);
*dbCtype = TextDatumGetCString(datum);
}
- if (dbIculocale)
+ if (dbLocale)
{
- datum = SysCacheGetAttr(DATABASEOID, tuple, Anum_pg_database_daticulocale, &isnull);
+ datum = SysCacheGetAttr(DATABASEOID, tuple, Anum_pg_database_datlocale, &isnull);
if (isnull)
- *dbIculocale = NULL;
+ *dbLocale = NULL;
else
- *dbIculocale = TextDatumGetCString(datum);
+ *dbLocale = TextDatumGetCString(datum);
}
if (dbIcurules)
{
diff --git a/src/backend/utils/adt/pg_locale.c b/src/backend/utils/adt/pg_locale.c
index 1dee4622d6..9d8634ec25 100644
--- a/src/backend/utils/adt/pg_locale.c
+++ b/src/backend/utils/adt/pg_locale.c
@@ -1606,7 +1606,7 @@ pg_newlocale_from_collation(Oid collid)
const char *iculocstr;
const char *icurules;
- datum = SysCacheGetAttrNotNull(COLLOID, tp, Anum_pg_collation_colliculocale);
+ datum = SysCacheGetAttrNotNull(COLLOID, tp, Anum_pg_collation_colllocale);
iculocstr = TextDatumGetCString(datum);
datum = SysCacheGetAttr(COLLOID, tp, Anum_pg_collation_collicurules, &isnull);
@@ -1627,7 +1627,7 @@ pg_newlocale_from_collation(Oid collid)
collversionstr = TextDatumGetCString(datum);
- datum = SysCacheGetAttrNotNull(COLLOID, tp, collform->collprovider == COLLPROVIDER_ICU ? Anum_pg_collation_colliculocale : Anum_pg_collation_collcollate);
+ datum = SysCacheGetAttrNotNull(COLLOID, tp, collform->collprovider == COLLPROVIDER_ICU ? Anum_pg_collation_colllocale : Anum_pg_collation_collcollate);
actual_versionstr = get_collation_actual_version(collform->collprovider,
TextDatumGetCString(datum));
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 552cf9d950..702bad6d8a 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -427,7 +427,7 @@ CheckMyDatabase(const char *name, bool am_superuser, bool override_allow_connect
{
char *icurules;
- datum = SysCacheGetAttrNotNull(DATABASEOID, tup, Anum_pg_database_daticulocale);
+ datum = SysCacheGetAttrNotNull(DATABASEOID, tup, Anum_pg_database_datlocale);
iculocale = TextDatumGetCString(datum);
datum = SysCacheGetAttr(DATABASEOID, tup, Anum_pg_database_daticurules, &isnull);
diff --git a/src/bin/initdb/initdb.c b/src/bin/initdb/initdb.c
index e68b40d2b5..eea40b930f 100644
--- a/src/bin/initdb/initdb.c
+++ b/src/bin/initdb/initdb.c
@@ -145,7 +145,8 @@ static char *lc_numeric = NULL;
static char *lc_time = NULL;
static char *lc_messages = NULL;
static char locale_provider = COLLPROVIDER_LIBC;
-static char *icu_locale = NULL;
+static char *datlocale = NULL;
+static bool icu_locale_specified = false;
static char *icu_rules = NULL;
static const char *default_text_search_config = NULL;
static char *username = NULL;
@@ -1523,8 +1524,8 @@ bootstrap_template1(void)
bki_lines = replace_token(bki_lines, "LC_CTYPE",
escape_quotes_bki(lc_ctype));
- bki_lines = replace_token(bki_lines, "ICU_LOCALE",
- icu_locale ? escape_quotes_bki(icu_locale) : "_null_");
+ bki_lines = replace_token(bki_lines, "DATLOCALE",
+ datlocale ? escape_quotes_bki(datlocale) : "_null_");
bki_lines = replace_token(bki_lines, "ICU_RULES",
icu_rules ? escape_quotes_bki(icu_rules) : "_null_");
@@ -2371,8 +2372,8 @@ setlocales(void)
lc_monetary = locale;
if (!lc_messages)
lc_messages = locale;
- if (!icu_locale && locale_provider == COLLPROVIDER_ICU)
- icu_locale = locale;
+ if (!datlocale && locale_provider != COLLPROVIDER_LIBC)
+ datlocale = locale;
}
/*
@@ -2398,22 +2399,21 @@ setlocales(void)
lc_messages = canonname;
#endif
+ if (locale_provider != COLLPROVIDER_LIBC && datlocale == NULL)
+ pg_fatal("locale must be specified unless provider is libc");
+
if (locale_provider == COLLPROVIDER_ICU)
{
char *langtag;
- /* acquire default locale from the environment, if not specified */
- if (icu_locale == NULL)
- pg_fatal("ICU locale must be specified");
-
/* canonicalize to a language tag */
- langtag = icu_language_tag(icu_locale);
+ langtag = icu_language_tag(datlocale);
printf(_("Using language tag \"%s\" for ICU locale \"%s\".\n"),
- langtag, icu_locale);
- pg_free(icu_locale);
- icu_locale = langtag;
+ langtag, datlocale);
+ pg_free(datlocale);
+ datlocale = langtag;
- icu_validate_locale(icu_locale);
+ icu_validate_locale(datlocale);
/*
* In supported builds, the ICU locale ID will be opened during
@@ -2607,14 +2607,14 @@ setup_locale_encoding(void)
strcmp(lc_ctype, lc_numeric) == 0 &&
strcmp(lc_ctype, lc_monetary) == 0 &&
strcmp(lc_ctype, lc_messages) == 0 &&
- (!icu_locale || strcmp(lc_ctype, icu_locale) == 0))
+ (!datlocale || strcmp(lc_ctype, datlocale) == 0))
printf(_("The database cluster will be initialized with locale \"%s\".\n"), lc_ctype);
else
{
printf(_("The database cluster will be initialized with this locale configuration:\n"));
- printf(_(" provider: %s\n"), collprovider_name(locale_provider));
- if (icu_locale)
- printf(_(" ICU locale: %s\n"), icu_locale);
+ printf(_(" default collation provider: %s\n"), collprovider_name(locale_provider));
+ if (locale_provider != COLLPROVIDER_LIBC)
+ printf(_(" default collation locale: %s\n"), datlocale);
printf(_(" LC_COLLATE: %s\n"
" LC_CTYPE: %s\n"
" LC_MESSAGES: %s\n"
@@ -3285,7 +3285,8 @@ main(int argc, char *argv[])
pg_fatal("unrecognized locale provider: %s", optarg);
break;
case 16:
- icu_locale = pg_strdup(optarg);
+ datlocale = pg_strdup(optarg);
+ icu_locale_specified = true;
break;
case 17:
icu_rules = pg_strdup(optarg);
@@ -3320,7 +3321,7 @@ main(int argc, char *argv[])
exit(1);
}
- if (icu_locale && locale_provider != COLLPROVIDER_ICU)
+ if (icu_locale_specified && locale_provider != COLLPROVIDER_ICU)
pg_fatal("%s cannot be specified unless locale provider \"%s\" is chosen",
"--icu-locale", "icu");
diff --git a/src/bin/initdb/t/001_initdb.pl b/src/bin/initdb/t/001_initdb.pl
index 45f96cd8bb..7606db1987 100644
--- a/src/bin/initdb/t/001_initdb.pl
+++ b/src/bin/initdb/t/001_initdb.pl
@@ -117,7 +117,7 @@ if ($ENV{with_icu} eq 'yes')
{
command_fails_like(
[ 'initdb', '--no-sync', '--locale-provider=icu', "$tempdir/data2" ],
- qr/initdb: error: ICU locale must be specified/,
+ qr/initdb: error: locale must be specified unless provider is libc/,
'locale provider ICU requires --icu-locale');
command_ok(
@@ -138,7 +138,7 @@ if ($ENV{with_icu} eq 'yes')
'--lc-monetary=C', '--lc-time=C',
"$tempdir/data4"
],
- qr/^\s+ICU locale:\s+und\n/ms,
+ qr/^\s+default collation locale:\s+und\n/ms,
'options --locale-provider=icu --locale=und --lc-*=C');
command_fails_like(
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 8c0b5486b9..3d829c99d7 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -2961,7 +2961,7 @@ dumpDatabase(Archive *fout)
i_datlocprovider,
i_collate,
i_ctype,
- i_daticulocale,
+ i_datlocale,
i_daticurules,
i_frozenxid,
i_minmxid,
@@ -2980,7 +2980,7 @@ dumpDatabase(Archive *fout)
*datlocprovider,
*collate,
*ctype,
- *iculocale,
+ *locale,
*icurules,
*datistemplate,
*datconnlimit,
@@ -3004,10 +3004,12 @@ dumpDatabase(Archive *fout)
appendPQExpBufferStr(dbQry, "datminmxid, ");
else
appendPQExpBufferStr(dbQry, "0 AS datminmxid, ");
- if (fout->remoteVersion >= 150000)
- appendPQExpBufferStr(dbQry, "datlocprovider, daticulocale, datcollversion, ");
+ if (fout->remoteVersion >= 170000)
+ appendPQExpBufferStr(dbQry, "datlocprovider, datlocale, datcollversion, ");
+ else if (fout->remoteVersion >= 150000)
+ appendPQExpBufferStr(dbQry, "datlocprovider, daticulocale AS datlocale, datcollversion, ");
else
- appendPQExpBufferStr(dbQry, "'c' AS datlocprovider, NULL AS daticulocale, NULL AS datcollversion, ");
+ appendPQExpBufferStr(dbQry, "'c' AS datlocprovider, NULL AS datlocale, NULL AS datcollversion, ");
if (fout->remoteVersion >= 160000)
appendPQExpBufferStr(dbQry, "daticurules, ");
else
@@ -3028,7 +3030,7 @@ dumpDatabase(Archive *fout)
i_datlocprovider = PQfnumber(res, "datlocprovider");
i_collate = PQfnumber(res, "datcollate");
i_ctype = PQfnumber(res, "datctype");
- i_daticulocale = PQfnumber(res, "daticulocale");
+ i_datlocale = PQfnumber(res, "datlocale");
i_daticurules = PQfnumber(res, "daticurules");
i_frozenxid = PQfnumber(res, "datfrozenxid");
i_minmxid = PQfnumber(res, "datminmxid");
@@ -3047,10 +3049,10 @@ dumpDatabase(Archive *fout)
datlocprovider = PQgetvalue(res, 0, i_datlocprovider);
collate = PQgetvalue(res, 0, i_collate);
ctype = PQgetvalue(res, 0, i_ctype);
- if (!PQgetisnull(res, 0, i_daticulocale))
- iculocale = PQgetvalue(res, 0, i_daticulocale);
+ if (!PQgetisnull(res, 0, i_datlocale))
+ locale = PQgetvalue(res, 0, i_datlocale);
else
- iculocale = NULL;
+ locale = NULL;
if (!PQgetisnull(res, 0, i_daticurules))
icurules = PQgetvalue(res, 0, i_daticurules);
else
@@ -3097,29 +3099,36 @@ dumpDatabase(Archive *fout)
pg_fatal("unrecognized locale provider: %s",
datlocprovider);
- if (strlen(collate) > 0 && strcmp(collate, ctype) == 0)
+ if (!locale && datlocprovider[0] != 'c')
+ pg_log_warning("database '%s' with provider '%s' missing datlocale",
+ datname, datlocprovider);
+
+ if (locale && datlocprovider[0] == 'c')
+ pg_log_warning("database '%s' with provider 'c' has non-NULL locale '%s'",
+ datname, locale);
+
+ /* if collate and ctype are equal, and locale is NULL, use LOCALE */
+ if (!locale && strlen(collate) > 0 && strcmp(collate, ctype) == 0)
+ locale = collate;
+
+ /* output LC_COLLATE and LC_CTYPE if different from LOCALE */
+ if (strlen(collate) > 0 && (!locale || strcmp(collate, locale) != 0))
{
- appendPQExpBufferStr(creaQry, " LOCALE = ");
+ appendPQExpBufferStr(creaQry, " LC_COLLATE = ");
appendStringLiteralAH(creaQry, collate, fout);
}
- else
+ if (strlen(ctype) > 0 && (!locale || strcmp(ctype, locale) != 0))
{
- if (strlen(collate) > 0)
- {
- appendPQExpBufferStr(creaQry, " LC_COLLATE = ");
- appendStringLiteralAH(creaQry, collate, fout);
- }
- if (strlen(ctype) > 0)
- {
- appendPQExpBufferStr(creaQry, " LC_CTYPE = ");
- appendStringLiteralAH(creaQry, ctype, fout);
- }
+ appendPQExpBufferStr(creaQry, " LC_CTYPE = ");
+ appendStringLiteralAH(creaQry, ctype, fout);
}
- if (iculocale)
+
+ if (locale)
{
- appendPQExpBufferStr(creaQry, " ICU_LOCALE = ");
- appendStringLiteralAH(creaQry, iculocale, fout);
+ appendPQExpBufferStr(creaQry, " LOCALE = ");
+ appendStringLiteralAH(creaQry, locale, fout);
}
+
if (icurules)
{
appendPQExpBufferStr(creaQry, " ICU_RULES = ");
@@ -13552,12 +13561,12 @@ dumpCollation(Archive *fout, const CollInfo *collinfo)
int i_collisdeterministic;
int i_collcollate;
int i_collctype;
- int i_colliculocale;
+ int i_colllocale;
int i_collicurules;
const char *collprovider;
const char *collcollate;
const char *collctype;
- const char *colliculocale;
+ const char *colllocale;
const char *collicurules;
/* Do nothing in data-only dump */
@@ -13589,12 +13598,15 @@ dumpCollation(Archive *fout, const CollInfo *collinfo)
appendPQExpBufferStr(query,
"true AS collisdeterministic, ");
- if (fout->remoteVersion >= 150000)
+ if (fout->remoteVersion >= 170000)
+ appendPQExpBufferStr(query,
+ "colllocale, ");
+ else if (fout->remoteVersion >= 150000)
appendPQExpBufferStr(query,
- "colliculocale, ");
+ "colliculocale AS colllocale, ");
else
appendPQExpBufferStr(query,
- "NULL AS colliculocale, ");
+ "NULL AS colllocale, ");
if (fout->remoteVersion >= 160000)
appendPQExpBufferStr(query,
@@ -13616,7 +13628,7 @@ dumpCollation(Archive *fout, const CollInfo *collinfo)
i_collisdeterministic = PQfnumber(res, "collisdeterministic");
i_collcollate = PQfnumber(res, "collcollate");
i_collctype = PQfnumber(res, "collctype");
- i_colliculocale = PQfnumber(res, "colliculocale");
+ i_colllocale = PQfnumber(res, "colllocale");
i_collicurules = PQfnumber(res, "collicurules");
collprovider = PQgetvalue(res, 0, i_collprovider);
@@ -13643,10 +13655,10 @@ dumpCollation(Archive *fout, const CollInfo *collinfo)
collctype = NULL;
}
- if (!PQgetisnull(res, 0, i_colliculocale))
- colliculocale = PQgetvalue(res, 0, i_colliculocale);
+ if (!PQgetisnull(res, 0, i_colllocale))
+ colllocale = PQgetvalue(res, 0, i_colllocale);
else
- colliculocale = NULL;
+ colllocale = NULL;
if (!PQgetisnull(res, 0, i_collicurules))
collicurules = PQgetvalue(res, 0, i_collicurules);
@@ -13676,7 +13688,7 @@ dumpCollation(Archive *fout, const CollInfo *collinfo)
if (collprovider[0] == 'd')
{
- if (collcollate || collctype || colliculocale || collicurules)
+ if (collcollate || collctype || colllocale || collicurules)
pg_log_warning("invalid collation \"%s\"", qcollname);
/* no locale -- the default collation cannot be reloaded anyway */
@@ -13685,16 +13697,16 @@ dumpCollation(Archive *fout, const CollInfo *collinfo)
{
if (fout->remoteVersion >= 150000)
{
- if (collcollate || collctype || !colliculocale)
+ if (collcollate || collctype || !colllocale)
pg_log_warning("invalid collation \"%s\"", qcollname);
appendPQExpBufferStr(q, ", locale = ");
- appendStringLiteralAH(q, colliculocale ? colliculocale : "",
+ appendStringLiteralAH(q, colllocale ? colllocale : "",
fout);
}
else
{
- if (!collcollate || !collctype || colliculocale ||
+ if (!collcollate || !collctype || colllocale ||
strcmp(collcollate, collctype) != 0)
pg_log_warning("invalid collation \"%s\"", qcollname);
@@ -13710,7 +13722,7 @@ dumpCollation(Archive *fout, const CollInfo *collinfo)
}
else if (collprovider[0] == 'c')
{
- if (colliculocale || collicurules || !collcollate || !collctype)
+ if (colllocale || collicurules || !collcollate || !collctype)
pg_log_warning("invalid collation \"%s\"", qcollname);
if (collcollate && collctype && strcmp(collcollate, collctype) == 0)
diff --git a/src/bin/pg_upgrade/info.c b/src/bin/pg_upgrade/info.c
index 4878aa22bf..38d52f5c64 100644
--- a/src/bin/pg_upgrade/info.c
+++ b/src/bin/pg_upgrade/info.c
@@ -323,18 +323,24 @@ get_template0_info(ClusterInfo *cluster)
int i_datlocprovider;
int i_datcollate;
int i_datctype;
- int i_daticulocale;
+ int i_datlocale;
- if (GET_MAJOR_VERSION(cluster->major_version) >= 1500)
+ if (GET_MAJOR_VERSION(cluster->major_version) >= 1700)
dbres = executeQueryOrDie(conn,
"SELECT encoding, datlocprovider, "
- " datcollate, datctype, daticulocale "
+ " datcollate, datctype, datlocale "
+ "FROM pg_catalog.pg_database "
+ "WHERE datname='template0'");
+ else if (GET_MAJOR_VERSION(cluster->major_version) >= 1500)
+ dbres = executeQueryOrDie(conn,
+ "SELECT encoding, datlocprovider, "
+ " datcollate, datctype, daticulocale AS datlocale"
"FROM pg_catalog.pg_database "
"WHERE datname='template0'");
else
dbres = executeQueryOrDie(conn,
"SELECT encoding, 'c' AS datlocprovider, "
- " datcollate, datctype, NULL AS daticulocale "
+ " datcollate, datctype, NULL AS datlocale "
"FROM pg_catalog.pg_database "
"WHERE datname='template0'");
@@ -348,16 +354,16 @@ get_template0_info(ClusterInfo *cluster)
i_datlocprovider = PQfnumber(dbres, "datlocprovider");
i_datcollate = PQfnumber(dbres, "datcollate");
i_datctype = PQfnumber(dbres, "datctype");
- i_daticulocale = PQfnumber(dbres, "daticulocale");
+ i_datlocale = PQfnumber(dbres, "datlocale");
locale->db_encoding = atoi(PQgetvalue(dbres, 0, i_datencoding));
locale->db_collprovider = PQgetvalue(dbres, 0, i_datlocprovider)[0];
locale->db_collate = pg_strdup(PQgetvalue(dbres, 0, i_datcollate));
locale->db_ctype = pg_strdup(PQgetvalue(dbres, 0, i_datctype));
- if (PQgetisnull(dbres, 0, i_daticulocale))
- locale->db_iculocale = NULL;
+ if (PQgetisnull(dbres, 0, i_datlocale))
+ locale->db_locale = NULL;
else
- locale->db_iculocale = pg_strdup(PQgetvalue(dbres, 0, i_daticulocale));
+ locale->db_locale = pg_strdup(PQgetvalue(dbres, 0, i_datlocale));
cluster->template0 = locale;
@@ -387,12 +393,15 @@ get_db_infos(ClusterInfo *cluster)
snprintf(query, sizeof(query),
"SELECT d.oid, d.datname, d.encoding, d.datcollate, d.datctype, ");
- if (GET_MAJOR_VERSION(cluster->major_version) < 1500)
+ if (GET_MAJOR_VERSION(cluster->major_version) >= 1700)
+ snprintf(query + strlen(query), sizeof(query) - strlen(query),
+ "datlocprovider, datlocale, ");
+ else if (GET_MAJOR_VERSION(cluster->major_version) >= 1500)
snprintf(query + strlen(query), sizeof(query) - strlen(query),
- "'c' AS datlocprovider, NULL AS daticulocale, ");
+ "datlocprovider, daticulocale AS datlocale, ");
else
snprintf(query + strlen(query), sizeof(query) - strlen(query),
- "datlocprovider, daticulocale, ");
+ "'c' AS datlocprovider, NULL AS datlocale, ");
snprintf(query + strlen(query), sizeof(query) - strlen(query),
"pg_catalog.pg_tablespace_location(t.oid) AS spclocation "
"FROM pg_catalog.pg_database d "
diff --git a/src/bin/pg_upgrade/pg_upgrade.c b/src/bin/pg_upgrade/pg_upgrade.c
index 3960af4036..a7013d200d 100644
--- a/src/bin/pg_upgrade/pg_upgrade.c
+++ b/src/bin/pg_upgrade/pg_upgrade.c
@@ -391,7 +391,7 @@ setup(char *argv0, bool *live_check)
* Copy locale and encoding information into the new cluster's template0.
*
* We need to copy the encoding, datlocprovider, datcollate, datctype, and
- * daticulocale. We don't need datcollversion because that's never set for
+ * datlocale. We don't need datcollversion because that's never set for
* template0.
*/
static void
@@ -400,7 +400,7 @@ set_locale_and_encoding(void)
PGconn *conn_new_template1;
char *datcollate_literal;
char *datctype_literal;
- char *daticulocale_literal = NULL;
+ char *datlocale_literal = NULL;
DbLocaleInfo *locale = old_cluster.template0;
prep_status("Setting locale and encoding for new cluster");
@@ -414,15 +414,29 @@ set_locale_and_encoding(void)
datctype_literal = PQescapeLiteral(conn_new_template1,
locale->db_ctype,
strlen(locale->db_ctype));
- if (locale->db_iculocale)
- daticulocale_literal = PQescapeLiteral(conn_new_template1,
- locale->db_iculocale,
- strlen(locale->db_iculocale));
+ if (locale->db_locale)
+ datlocale_literal = PQescapeLiteral(conn_new_template1,
+ locale->db_locale,
+ strlen(locale->db_locale));
else
- daticulocale_literal = pg_strdup("NULL");
+ datlocale_literal = pg_strdup("NULL");
/* update template0 in new cluster */
- if (GET_MAJOR_VERSION(new_cluster.major_version) >= 1500)
+ if (GET_MAJOR_VERSION(new_cluster.major_version) >= 1700)
+ PQclear(executeQueryOrDie(conn_new_template1,
+ "UPDATE pg_catalog.pg_database "
+ " SET encoding = %d, "
+ " datlocprovider = '%c', "
+ " datcollate = %s, "
+ " datctype = %s, "
+ " datlocale = %s "
+ " WHERE datname = 'template0' ",
+ locale->db_encoding,
+ locale->db_collprovider,
+ datcollate_literal,
+ datctype_literal,
+ datlocale_literal));
+ else if (GET_MAJOR_VERSION(new_cluster.major_version) >= 1500)
PQclear(executeQueryOrDie(conn_new_template1,
"UPDATE pg_catalog.pg_database "
" SET encoding = %d, "
@@ -435,7 +449,7 @@ set_locale_and_encoding(void)
locale->db_collprovider,
datcollate_literal,
datctype_literal,
- daticulocale_literal));
+ datlocale_literal));
else
PQclear(executeQueryOrDie(conn_new_template1,
"UPDATE pg_catalog.pg_database "
@@ -449,7 +463,7 @@ set_locale_and_encoding(void)
PQfreemem(datcollate_literal);
PQfreemem(datctype_literal);
- PQfreemem(daticulocale_literal);
+ PQfreemem(datlocale_literal);
PQfinish(conn_new_template1);
diff --git a/src/bin/pg_upgrade/pg_upgrade.h b/src/bin/pg_upgrade/pg_upgrade.h
index a710f325de..81eb60cb2e 100644
--- a/src/bin/pg_upgrade/pg_upgrade.h
+++ b/src/bin/pg_upgrade/pg_upgrade.h
@@ -205,7 +205,7 @@ typedef struct
char *db_collate;
char *db_ctype;
char db_collprovider;
- char *db_iculocale;
+ char *db_locale;
int db_encoding;
} DbLocaleInfo;
diff --git a/src/bin/pg_upgrade/t/002_pg_upgrade.pl b/src/bin/pg_upgrade/t/002_pg_upgrade.pl
index e5f57e550a..6f359d72ce 100644
--- a/src/bin/pg_upgrade/t/002_pg_upgrade.pl
+++ b/src/bin/pg_upgrade/t/002_pg_upgrade.pl
@@ -113,11 +113,18 @@ my $original_provider = "c";
my $original_locale = "C";
my $original_iculocale = "";
my $provider_field = "'c' AS datlocprovider";
-my $iculocale_field = "NULL AS daticulocale";
-if ($oldnode->pg_version >= 15 && $ENV{with_icu} eq 'yes')
+my $iculocale_field = "NULL AS datlocale";
+if (int($oldnode->pg_version) >= 15 && $ENV{with_icu} eq 'yes')
{
$provider_field = "datlocprovider";
- $iculocale_field = "daticulocale";
+ if (int($oldnode->pg_version) >= 17)
+ {
+ $iculocale_field = "datlocale";
+ }
+ else
+ {
+ $iculocale_field = "daticulocale AS datlocale";
+ }
$original_provider = "i";
$original_iculocale = "fr-CA";
}
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index 5077e7b358..1122843715 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -937,14 +937,18 @@ listAllDbs(const char *pattern, bool verbose)
" d.datctype as \"%s\",\n",
gettext_noop("Collate"),
gettext_noop("Ctype"));
- if (pset.sversion >= 150000)
+ if (pset.sversion >= 170000)
+ appendPQExpBuffer(&buf,
+ " d.datlocale as \"%s\",\n",
+ gettext_noop("Locale"));
+ else if (pset.sversion >= 150000)
appendPQExpBuffer(&buf,
" d.daticulocale as \"%s\",\n",
- gettext_noop("ICU Locale"));
+ gettext_noop("Locale"));
else
appendPQExpBuffer(&buf,
" NULL as \"%s\",\n",
- gettext_noop("ICU Locale"));
+ gettext_noop("Locale"));
if (pset.sversion >= 160000)
appendPQExpBuffer(&buf,
" d.daticurules as \"%s\",\n",
@@ -4975,14 +4979,18 @@ listCollations(const char *pattern, bool verbose, bool showSystem)
gettext_noop("Collate"),
gettext_noop("Ctype"));
- if (pset.sversion >= 150000)
+ if (pset.sversion >= 170000)
+ appendPQExpBuffer(&buf,
+ " c.colllocale AS \"%s\",\n",
+ gettext_noop("Locale"));
+ else if (pset.sversion >= 150000)
appendPQExpBuffer(&buf,
" c.colliculocale AS \"%s\",\n",
- gettext_noop("ICU Locale"));
+ gettext_noop("Locale"));
else
appendPQExpBuffer(&buf,
" c.collcollate AS \"%s\",\n",
- gettext_noop("ICU Locale"));
+ gettext_noop("Locale"));
if (pset.sversion >= 160000)
appendPQExpBuffer(&buf,
diff --git a/src/include/catalog/pg_collation.dat b/src/include/catalog/pg_collation.dat
index b6a69d1d42..2c112cd6bc 100644
--- a/src/include/catalog/pg_collation.dat
+++ b/src/include/catalog/pg_collation.dat
@@ -29,6 +29,6 @@
{ oid => '963',
descr => 'sorts using the Unicode Collation Algorithm with default settings',
collname => 'unicode', collprovider => 'i', collencoding => '-1',
- colliculocale => 'und' },
+ colllocale => 'und' },
]
diff --git a/src/include/catalog/pg_collation.h b/src/include/catalog/pg_collation.h
index 2d7dcde183..d357c89ae6 100644
--- a/src/include/catalog/pg_collation.h
+++ b/src/include/catalog/pg_collation.h
@@ -42,7 +42,7 @@ CATALOG(pg_collation,3456,CollationRelationId)
#ifdef CATALOG_VARLEN /* variable-length fields start here */
text collcollate BKI_DEFAULT(_null_); /* LC_COLLATE setting */
text collctype BKI_DEFAULT(_null_); /* LC_CTYPE setting */
- text colliculocale BKI_DEFAULT(_null_); /* ICU locale ID */
+ text colllocale BKI_DEFAULT(_null_); /* locale ID */
text collicurules BKI_DEFAULT(_null_); /* ICU collation rules */
text collversion BKI_DEFAULT(_null_); /* provider-dependent
* version of collation
@@ -91,7 +91,7 @@ extern Oid CollationCreate(const char *collname, Oid collnamespace,
bool collisdeterministic,
int32 collencoding,
const char *collcollate, const char *collctype,
- const char *colliculocale,
+ const char *colllocale,
const char *collicurules,
const char *collversion,
bool if_not_exists,
diff --git a/src/include/catalog/pg_database.dat b/src/include/catalog/pg_database.dat
index 8d91e3bf8d..0b54609d66 100644
--- a/src/include/catalog/pg_database.dat
+++ b/src/include/catalog/pg_database.dat
@@ -18,7 +18,7 @@
datlocprovider => 'LOCALE_PROVIDER', datistemplate => 't',
datallowconn => 't', dathasloginevt => 'f', datconnlimit => '-1', datfrozenxid => '0',
datminmxid => '1', dattablespace => 'pg_default', datcollate => 'LC_COLLATE',
- datctype => 'LC_CTYPE', daticulocale => 'ICU_LOCALE',
+ datctype => 'LC_CTYPE', datlocale => 'DATLOCALE',
daticurules => 'ICU_RULES', datacl => '_null_' },
]
diff --git a/src/include/catalog/pg_database.h b/src/include/catalog/pg_database.h
index 3e50a57004..c659b5165a 100644
--- a/src/include/catalog/pg_database.h
+++ b/src/include/catalog/pg_database.h
@@ -75,7 +75,7 @@ CATALOG(pg_database,1262,DatabaseRelationId) BKI_SHARED_RELATION BKI_ROWTYPE_OID
text datctype BKI_FORCE_NOT_NULL;
/* ICU locale ID */
- text daticulocale;
+ text datlocale;
/* ICU collation rules */
text daticurules;
diff --git a/src/test/regress/expected/collate.icu.utf8.out b/src/test/regress/expected/collate.icu.utf8.out
index 7a05c75967..8ca93f4dea 100644
--- a/src/test/regress/expected/collate.icu.utf8.out
+++ b/src/test/regress/expected/collate.icu.utf8.out
@@ -1024,7 +1024,7 @@ SET icu_validation_level = disabled;
do $$
BEGIN
EXECUTE 'CREATE COLLATION test0 (provider = icu, locale = ' ||
- quote_literal((SELECT CASE WHEN datlocprovider='i' THEN daticulocale ELSE datcollate END FROM pg_database WHERE datname = current_database())) || ');';
+ quote_literal((SELECT CASE WHEN datlocprovider='i' THEN datlocale ELSE datcollate END FROM pg_database WHERE datname = current_database())) || ');';
END
$$;
CREATE COLLATION test0 FROM "C"; -- fail, duplicate name
@@ -1032,7 +1032,7 @@ ERROR: collation "test0" already exists
do $$
BEGIN
EXECUTE 'CREATE COLLATION test1 (provider = icu, locale = ' ||
- quote_literal((SELECT CASE WHEN datlocprovider='i' THEN daticulocale ELSE datcollate END FROM pg_database WHERE datname = current_database())) || ');';
+ quote_literal((SELECT CASE WHEN datlocprovider='i' THEN datlocale ELSE datcollate END FROM pg_database WHERE datname = current_database())) || ');';
END
$$;
RESET icu_validation_level;
diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out
index 5d61e4c7bb..53d492c750 100644
--- a/src/test/regress/expected/psql.out
+++ b/src/test/regress/expected/psql.out
@@ -6220,9 +6220,9 @@ List of schemas
(0 rows)
\dO "no.such.collation"
- List of collations
- Schema | Name | Provider | Collate | Ctype | ICU Locale | ICU Rules | Deterministic?
---------+------+----------+---------+-------+------------+-----------+----------------
+ List of collations
+ Schema | Name | Provider | Collate | Ctype | Locale | ICU Rules | Deterministic?
+--------+------+----------+---------+-------+--------+-----------+----------------
(0 rows)
\dp "no.such.access.privilege"
@@ -6409,9 +6409,9 @@ cross-database references are not implemented: "no.such.schema"."no.such.languag
(0 rows)
\dO "no.such.schema"."no.such.collation"
- List of collations
- Schema | Name | Provider | Collate | Ctype | ICU Locale | ICU Rules | Deterministic?
---------+------+----------+---------+-------+------------+-----------+----------------
+ List of collations
+ Schema | Name | Provider | Collate | Ctype | Locale | ICU Rules | Deterministic?
+--------+------+----------+---------+-------+--------+-----------+----------------
(0 rows)
\dp "no.such.schema"."no.such.access.privilege"
@@ -6552,9 +6552,9 @@ List of text search templates
(0 rows)
\dO regression."no.such.schema"."no.such.collation"
- List of collations
- Schema | Name | Provider | Collate | Ctype | ICU Locale | ICU Rules | Deterministic?
---------+------+----------+---------+-------+------------+-----------+----------------
+ List of collations
+ Schema | Name | Provider | Collate | Ctype | Locale | ICU Rules | Deterministic?
+--------+------+----------+---------+-------+--------+-----------+----------------
(0 rows)
\dp regression."no.such.schema"."no.such.access.privilege"
diff --git a/src/test/regress/sql/collate.icu.utf8.sql b/src/test/regress/sql/collate.icu.utf8.sql
index 3db9e25913..03837de846 100644
--- a/src/test/regress/sql/collate.icu.utf8.sql
+++ b/src/test/regress/sql/collate.icu.utf8.sql
@@ -363,14 +363,14 @@ SET icu_validation_level = disabled;
do $$
BEGIN
EXECUTE 'CREATE COLLATION test0 (provider = icu, locale = ' ||
- quote_literal((SELECT CASE WHEN datlocprovider='i' THEN daticulocale ELSE datcollate END FROM pg_database WHERE datname = current_database())) || ');';
+ quote_literal((SELECT CASE WHEN datlocprovider='i' THEN datlocale ELSE datcollate END FROM pg_database WHERE datname = current_database())) || ');';
END
$$;
CREATE COLLATION test0 FROM "C"; -- fail, duplicate name
do $$
BEGIN
EXECUTE 'CREATE COLLATION test1 (provider = icu, locale = ' ||
- quote_literal((SELECT CASE WHEN datlocprovider='i' THEN daticulocale ELSE datcollate END FROM pg_database WHERE datname = current_database())) || ');';
+ quote_literal((SELECT CASE WHEN datlocprovider='i' THEN datlocale ELSE datcollate END FROM pg_database WHERE datname = current_database())) || ');';
END
$$;
--
2.34.1
[text/x-patch] v15-0005-Introduce-collation-provider-builtin-for-C-and-C.patch (73.0K, ../../[email protected]/6-v15-0005-Introduce-collation-provider-builtin-for-C-and-C.patch)
download | inline diff:
From 5ffd8d5d3a327b86c8170a8125344e9b1aeb028d Mon Sep 17 00:00:00 2001
From: Jeff Davis <[email protected]>
Date: Mon, 1 May 2023 15:38:29 -0700
Subject: [PATCH v15 5/5] Introduce collation provider "builtin" for "C" and
"C.UTF-8".
The builtin "C" locale is equal (in semantics and implementation) to
the libc "C" locale.
The builtin "C.UTF-8" locale is especially useful. It provides a fast
memcmp-based collation (like "C") that supports abbrevated keys, while
also providing richer ctype semantics (upper/lower and regexes). The
semantics are derived from Unicode by building in lookup tables in the
same way as for text normalization. By using built-in semantics, the
behavior is stabilized within a Postgres major version, and also
matches the behavior of other built-in Unicode functionality, such as
normalization.
Discussion: https://postgr.es/m/[email protected]
Discussion: https://postgr.es/m/[email protected]
---
doc/src/sgml/charset.sgml | 88 +++++++--
doc/src/sgml/ref/create_collation.sgml | 11 +-
doc/src/sgml/ref/create_database.sgml | 8 +-
doc/src/sgml/ref/createdb.sgml | 2 +-
doc/src/sgml/ref/initdb.sgml | 7 +-
src/backend/catalog/pg_collation.c | 5 +-
src/backend/commands/collationcmds.c | 93 +++++++--
src/backend/commands/dbcommands.c | 123 +++++++++---
src/backend/regex/regc_pg_locale.c | 41 +++-
src/backend/utils/adt/formatting.c | 193 +++++++++++++++++++
src/backend/utils/adt/pg_locale.c | 130 +++++++++++--
src/backend/utils/init/postinit.c | 27 ++-
src/bin/initdb/initdb.c | 30 ++-
src/bin/initdb/t/001_initdb.pl | 55 ++++++
src/bin/pg_dump/pg_dump.c | 15 +-
src/bin/pg_upgrade/t/002_pg_upgrade.pl | 74 +++++--
src/bin/psql/describe.c | 4 +-
src/bin/scripts/createdb.c | 18 +-
src/bin/scripts/t/020_createdb.pl | 72 +++++++
src/include/catalog/pg_collation.dat | 9 +-
src/include/catalog/pg_collation.h | 3 +
src/include/utils/pg_locale.h | 7 +-
src/test/icu/t/010_database.pl | 18 +-
src/test/regress/expected/collate.out | 24 ++-
src/test/regress/expected/collate.utf8.out | 97 ++++++++++
src/test/regress/expected/collate.utf8_1.out | 8 +
src/test/regress/parallel_schedule | 4 +-
src/test/regress/sql/collate.sql | 10 +
src/test/regress/sql/collate.utf8.sql | 45 +++++
29 files changed, 1082 insertions(+), 139 deletions(-)
create mode 100644 src/test/regress/expected/collate.utf8.out
create mode 100644 src/test/regress/expected/collate.utf8_1.out
create mode 100644 src/test/regress/sql/collate.utf8.sql
diff --git a/doc/src/sgml/charset.sgml b/doc/src/sgml/charset.sgml
index 74783d148f..1553deea20 100644
--- a/doc/src/sgml/charset.sgml
+++ b/doc/src/sgml/charset.sgml
@@ -342,22 +342,14 @@ initdb --locale=sv_SE
<title>Locale Providers</title>
<para>
- <productname>PostgreSQL</productname> supports multiple <firstterm>locale
- providers</firstterm>. This specifies which library supplies the locale
- data. One standard provider name is <literal>libc</literal>, which uses
- the locales provided by the operating system C library. These are the
- locales used by most tools provided by the operating system. Another
- provider is <literal>icu</literal>, which uses the external
- ICU<indexterm><primary>ICU</primary></indexterm> library. ICU locales can
- only be used if support for ICU was configured when PostgreSQL was built.
+ A locale provider specifies which library defines the locale behavior for
+ collations and character classifications.
</para>
<para>
The commands and tools that select the locale settings, as described
- above, each have an option to select the locale provider. The examples
- shown earlier all use the <literal>libc</literal> provider, which is the
- default. Here is an example to initialize a database cluster using the
- ICU provider:
+ above, each have an option to select the locale provider. Here is an
+ example to initialize a database cluster using the ICU provider:
<programlisting>
initdb --locale-provider=icu --icu-locale=en
</programlisting>
@@ -370,12 +362,74 @@ initdb --locale-provider=icu --icu-locale=en
</para>
<para>
- Which locale provider to use depends on individual requirements. For most
- basic uses, either provider will give adequate results. For the libc
- provider, it depends on what the operating system offers; some operating
- systems are better than others. For advanced uses, ICU offers more locale
- variants and customization options.
+ Regardless of the locale provider, the operating system is still used to
+ provide some locale-aware behavior, such as messages (see <xref
+ linkend="guc-lc-messages"/>).
</para>
+
+ <para>
+ The available locale providers are listed below.
+ </para>
+
+ <sect3 id="locale-provider-builtin">
+ <title>Builtin</title>
+ <para>
+ The <literal>builtin</literal> provider uses built-in operations. Only
+ the <literal>C</literal> and <literal>C.UTF-8</literal> locales are
+ supported for this provider.
+ </para>
+ <para>
+ The <literal>C</literal> locale behavior is identical to the
+ <literal>C</literal> locale in the libc provider. When using this locale,
+ the behavior may depend on the database encoding.
+ </para>
+ <para>
+ The <literal>C.UTF-8</literal> locale is available only for when the
+ database encoding is <literal>UTF-8</literal>, and the behavior is based
+ on Unicode. The collation uses the code point values only. The regular
+ expression character classes are based on the "POSIX Compatible"
+ semantics, and the case mapping is the "simple" variant.
+ </para>
+ </sect3>
+ <sect3 id="locale-provider-icu">
+ <title>ICU</title>
+ <para>
+ The <literal>icu</literal> provider uses the external
+ ICU<indexterm><primary>ICU</primary></indexterm>
+ library. <productname>PostgreSQL</productname> must have been configured
+ with support.
+ </para>
+ <para>
+ ICU provides collation and character classification behavior that is
+ independent of the operating system and database encoding, which is
+ preferable if you expect to transition to other platforms without any
+ change in results. <literal>LC_COLLATE</literal> and
+ <literal>LC_CTYPE</literal> can be set independently of the ICU locale.
+ </para>
+ <note>
+ <para>
+ For the ICU provider, results may depend on the version of the ICU
+ library used, as it is updated to reflect changes in natural language
+ over time.
+ </para>
+ </note>
+ </sect3>
+ <sect3 id="locale-provider-libc">
+ <title>libc</title>
+ <para>
+ The <literal>libc</literal> provider uses the operating system's C
+ library. The collation and character classification behavior is
+ controlled by the settings <literal>LC_COLLATE</literal> and
+ <literal>LC_CTYPE</literal>, so they cannot be set independently.
+ </para>
+ <note>
+ <para>
+ The same locale name may have different behavior on different platforms
+ when using the libc provider.
+ </para>
+ </note>
+ </sect3>
+
</sect2>
<sect2 id="icu-locales">
diff --git a/doc/src/sgml/ref/create_collation.sgml b/doc/src/sgml/ref/create_collation.sgml
index 5cf9777764..85f18cbbe5 100644
--- a/doc/src/sgml/ref/create_collation.sgml
+++ b/doc/src/sgml/ref/create_collation.sgml
@@ -96,6 +96,11 @@ CREATE COLLATION [ IF NOT EXISTS ] <replaceable>name</replaceable> FROM <replace
<replaceable>locale</replaceable>, you cannot specify either of those
parameters.
</para>
+ <para>
+ If <replaceable>provider</replaceable> is <literal>builtin</literal>,
+ then <replaceable>locale</replaceable> must be specified and set to
+ either <literal>C</literal> or <literal>C.UTF-8</literal>.
+ </para>
</listitem>
</varlistentry>
@@ -129,9 +134,9 @@ CREATE COLLATION [ IF NOT EXISTS ] <replaceable>name</replaceable> FROM <replace
<listitem>
<para>
Specifies the provider to use for locale services associated with this
- collation. Possible values are
- <literal>icu</literal><indexterm><primary>ICU</primary></indexterm>
- (if the server was built with ICU support) or <literal>libc</literal>.
+ collation. Possible values are <literal>builtin</literal>,
+ <literal>icu</literal><indexterm><primary>ICU</primary></indexterm> (if
+ the server was built with ICU support) or <literal>libc</literal>.
<literal>libc</literal> is the default. See <xref
linkend="locale-providers"/> for details.
</para>
diff --git a/doc/src/sgml/ref/create_database.sgml b/doc/src/sgml/ref/create_database.sgml
index ce7317f81b..6dc3348d1b 100644
--- a/doc/src/sgml/ref/create_database.sgml
+++ b/doc/src/sgml/ref/create_database.sgml
@@ -162,6 +162,12 @@ CREATE DATABASE <replaceable class="parameter">name</replaceable>
linkend="create-database-lc-ctype"/>, or <xref
linkend="create-database-icu-locale"/> individually.
</para>
+ <para>
+ If <xref linkend="create-database-locale-provider"/> is
+ <literal>builtin</literal>, then <replaceable>locale</replaceable>
+ must be specified and set to either <literal>C</literal> or
+ <literal>C.UTF-8</literal>.
+ </para>
<tip>
<para>
The other locale settings <xref linkend="guc-lc-messages"/>, <xref
@@ -243,7 +249,7 @@ CREATE DATABASE <replaceable class="parameter">name</replaceable>
<listitem>
<para>
Specifies the provider to use for the default collation in this
- database. Possible values are
+ database. Possible values are <literal>builtin</literal>,
<literal>icu</literal><indexterm><primary>ICU</primary></indexterm>
(if the server was built with ICU support) or <literal>libc</literal>.
By default, the provider is the same as that of the <xref
diff --git a/doc/src/sgml/ref/createdb.sgml b/doc/src/sgml/ref/createdb.sgml
index e4647d5ce7..d3e815f659 100644
--- a/doc/src/sgml/ref/createdb.sgml
+++ b/doc/src/sgml/ref/createdb.sgml
@@ -171,7 +171,7 @@ PostgreSQL documentation
</varlistentry>
<varlistentry>
- <term><option>--locale-provider={<literal>libc</literal>|<literal>icu</literal>}</option></term>
+ <term><option>--locale-provider={<literal>builtin</literal>|<literal>libc</literal>|<literal>icu</literal>}</option></term>
<listitem>
<para>
Specifies the locale provider for the database's default collation.
diff --git a/doc/src/sgml/ref/initdb.sgml b/doc/src/sgml/ref/initdb.sgml
index d43c91575c..2192d017b3 100644
--- a/doc/src/sgml/ref/initdb.sgml
+++ b/doc/src/sgml/ref/initdb.sgml
@@ -286,6 +286,11 @@ PostgreSQL documentation
environment that <command>initdb</command> runs in. Locale
support is described in <xref linkend="locale"/>.
</para>
+ <para>
+ If <option>--locale-provider</option> is <literal>builtin</literal>,
+ <option>--locale</option> must be specified and set to
+ <literal>C</literal> or <literal>C.UTF-8</literal>.
+ </para>
</listitem>
</varlistentry>
@@ -315,7 +320,7 @@ PostgreSQL documentation
</varlistentry>
<varlistentry id="app-initdb-option-locale-provider">
- <term><option>--locale-provider={<literal>libc</literal>|<literal>icu</literal>}</option></term>
+ <term><option>--locale-provider={<literal>builtin</literal>|<literal>libc</literal>|<literal>icu</literal>}</option></term>
<listitem>
<para>
This option sets the locale provider for databases created in the new
diff --git a/src/backend/catalog/pg_collation.c b/src/backend/catalog/pg_collation.c
index b615d2fc7a..2144e11fe8 100644
--- a/src/backend/catalog/pg_collation.c
+++ b/src/backend/catalog/pg_collation.c
@@ -68,7 +68,10 @@ CollationCreate(const char *collname, Oid collnamespace,
Assert(collname);
Assert(collnamespace);
Assert(collowner);
- Assert((collcollate && collctype) || colllocale);
+ Assert((collprovider == COLLPROVIDER_LIBC &&
+ collcollate && collctype && !colllocale) ||
+ (collprovider != COLLPROVIDER_LIBC &&
+ !collcollate && !collctype && colllocale));
/*
* Make sure there is no existing collation of same name & encoding.
diff --git a/src/backend/commands/collationcmds.c b/src/backend/commands/collationcmds.c
index 19e61b3e5b..505b8ae86d 100644
--- a/src/backend/commands/collationcmds.c
+++ b/src/backend/commands/collationcmds.c
@@ -68,7 +68,7 @@ DefineCollation(ParseState *pstate, List *names, List *parameters, bool if_not_e
DefElem *versionEl = NULL;
char *collcollate;
char *collctype;
- char *colllocale;
+ const char *colllocale;
char *collicurules;
bool collisdeterministic;
int collencoding;
@@ -215,7 +215,9 @@ DefineCollation(ParseState *pstate, List *names, List *parameters, bool if_not_e
if (collproviderstr)
{
- if (pg_strcasecmp(collproviderstr, "icu") == 0)
+ if (pg_strcasecmp(collproviderstr, "builtin") == 0)
+ collprovider = COLLPROVIDER_BUILTIN;
+ else if (pg_strcasecmp(collproviderstr, "icu") == 0)
collprovider = COLLPROVIDER_ICU;
else if (pg_strcasecmp(collproviderstr, "libc") == 0)
collprovider = COLLPROVIDER_LIBC;
@@ -245,7 +247,17 @@ DefineCollation(ParseState *pstate, List *names, List *parameters, bool if_not_e
if (lcctypeEl)
collctype = defGetString(lcctypeEl);
- if (collprovider == COLLPROVIDER_LIBC)
+ if (collprovider == COLLPROVIDER_BUILTIN)
+ {
+ if (!colllocale)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_OBJECT_DEFINITION),
+ errmsg("parameter \"locale\" must be specified")));
+
+ colllocale = builtin_validate_locale(GetDatabaseEncoding(),
+ colllocale);
+ }
+ else if (collprovider == COLLPROVIDER_LIBC)
{
if (!collcollate)
ereport(ERROR,
@@ -305,7 +317,17 @@ DefineCollation(ParseState *pstate, List *names, List *parameters, bool if_not_e
(errcode(ERRCODE_INVALID_OBJECT_DEFINITION),
errmsg("ICU rules cannot be specified unless locale provider is ICU")));
- if (collprovider == COLLPROVIDER_ICU)
+ if (collprovider == COLLPROVIDER_BUILTIN)
+ {
+ /*
+ * Behavior may be different in different encodings, so set
+ * collencoding to the current database encoding. No validation is
+ * required, because the "builtin" provider is compatible with any
+ * encoding.
+ */
+ collencoding = GetDatabaseEncoding();
+ }
+ else if (collprovider == COLLPROVIDER_ICU)
{
#ifdef USE_ICU
/*
@@ -334,7 +356,18 @@ DefineCollation(ParseState *pstate, List *names, List *parameters, bool if_not_e
}
if (!collversion)
- collversion = get_collation_actual_version(collprovider, collprovider == COLLPROVIDER_ICU ? colllocale : collcollate);
+ {
+ const char *locale;
+
+ if (collprovider == COLLPROVIDER_ICU)
+ locale = colllocale;
+ else if (collprovider == COLLPROVIDER_LIBC)
+ locale = collcollate;
+ else
+ locale = NULL; /* COLLPROVIDER_BUILTIN */
+
+ collversion = get_collation_actual_version(collprovider, locale);
+ }
newoid = CollationCreate(collName,
collNamespace,
@@ -409,6 +442,7 @@ AlterCollation(AlterCollationStmt *stmt)
Form_pg_collation collForm;
Datum datum;
bool isnull;
+ char *locale;
char *oldversion;
char *newversion;
ObjectAddress address;
@@ -435,8 +469,20 @@ AlterCollation(AlterCollationStmt *stmt)
datum = SysCacheGetAttr(COLLOID, tup, Anum_pg_collation_collversion, &isnull);
oldversion = isnull ? NULL : TextDatumGetCString(datum);
- datum = SysCacheGetAttrNotNull(COLLOID, tup, collForm->collprovider == COLLPROVIDER_ICU ? Anum_pg_collation_colllocale : Anum_pg_collation_collcollate);
- newversion = get_collation_actual_version(collForm->collprovider, TextDatumGetCString(datum));
+ if (collForm->collprovider == COLLPROVIDER_ICU)
+ {
+ datum = SysCacheGetAttrNotNull(COLLOID, tup, Anum_pg_collation_colllocale);
+ locale = TextDatumGetCString(datum);
+ }
+ else if (collForm->collprovider == COLLPROVIDER_LIBC)
+ {
+ datum = SysCacheGetAttrNotNull(COLLOID, tup, Anum_pg_collation_collcollate);
+ locale = TextDatumGetCString(datum);
+ }
+ else
+ locale = NULL; /* COLLPROVIDER_BUILTIN */
+
+ newversion = get_collation_actual_version(collForm->collprovider, locale);
/* cannot change from NULL to non-NULL or vice versa */
if ((!oldversion && newversion) || (oldversion && !newversion))
@@ -500,11 +546,18 @@ pg_collation_actual_version(PG_FUNCTION_ARGS)
provider = ((Form_pg_database) GETSTRUCT(dbtup))->datlocprovider;
- datum = SysCacheGetAttrNotNull(DATABASEOID, dbtup,
- provider == COLLPROVIDER_ICU ?
- Anum_pg_database_datlocale : Anum_pg_database_datcollate);
-
- locale = TextDatumGetCString(datum);
+ if (provider == COLLPROVIDER_ICU)
+ {
+ datum = SysCacheGetAttrNotNull(DATABASEOID, dbtup, Anum_pg_database_datlocale);
+ locale = TextDatumGetCString(datum);
+ }
+ else if (provider == COLLPROVIDER_LIBC)
+ {
+ datum = SysCacheGetAttrNotNull(DATABASEOID, dbtup, Anum_pg_database_datcollate);
+ locale = TextDatumGetCString(datum);
+ }
+ else
+ locale = NULL; /* COLLPROVIDER_BUILTIN */
ReleaseSysCache(dbtup);
}
@@ -521,11 +574,19 @@ pg_collation_actual_version(PG_FUNCTION_ARGS)
provider = ((Form_pg_collation) GETSTRUCT(colltp))->collprovider;
Assert(provider != COLLPROVIDER_DEFAULT);
- datum = SysCacheGetAttrNotNull(COLLOID, colltp,
- provider == COLLPROVIDER_ICU ?
- Anum_pg_collation_colllocale : Anum_pg_collation_collcollate);
- locale = TextDatumGetCString(datum);
+ if (provider == COLLPROVIDER_ICU)
+ {
+ datum = SysCacheGetAttrNotNull(COLLOID, colltp, Anum_pg_collation_colllocale);
+ locale = TextDatumGetCString(datum);
+ }
+ else if (provider == COLLPROVIDER_LIBC)
+ {
+ datum = SysCacheGetAttrNotNull(COLLOID, colltp, Anum_pg_collation_collcollate);
+ locale = TextDatumGetCString(datum);
+ }
+ else
+ locale = NULL; /* COLLPROVIDER_BUILTIN */
ReleaseSysCache(colltp);
}
diff --git a/src/backend/commands/dbcommands.c b/src/backend/commands/dbcommands.c
index c8732be067..b04ff25562 100644
--- a/src/backend/commands/dbcommands.c
+++ b/src/backend/commands/dbcommands.c
@@ -698,6 +698,7 @@ createdb(ParseState *pstate, const CreatedbStmt *stmt)
DefElem *dtemplate = NULL;
DefElem *dencoding = NULL;
DefElem *dlocale = NULL;
+ DefElem *dbuiltinlocale = NULL;
DefElem *dcollate = NULL;
DefElem *dctype = NULL;
DefElem *diculocale = NULL;
@@ -713,7 +714,7 @@ createdb(ParseState *pstate, const CreatedbStmt *stmt)
const char *dbtemplate = NULL;
char *dbcollate = NULL;
char *dbctype = NULL;
- char *dblocale = NULL;
+ const char *dblocale = NULL;
char *dbicurules = NULL;
char dblocprovider = '\0';
char *canonname;
@@ -762,6 +763,12 @@ createdb(ParseState *pstate, const CreatedbStmt *stmt)
errorConflictingDefElem(defel, pstate);
dlocale = defel;
}
+ else if (strcmp(defel->defname, "builtin_locale") == 0)
+ {
+ if (dbuiltinlocale)
+ errorConflictingDefElem(defel, pstate);
+ dbuiltinlocale = defel;
+ }
else if (strcmp(defel->defname, "lc_collate") == 0)
{
if (dcollate)
@@ -897,7 +904,10 @@ createdb(ParseState *pstate, const CreatedbStmt *stmt)
{
dbcollate = defGetString(dlocale);
dbctype = defGetString(dlocale);
+ dblocale = defGetString(dlocale);
}
+ if (dbuiltinlocale && dbuiltinlocale->arg)
+ dblocale = defGetString(dbuiltinlocale);
if (dcollate && dcollate->arg)
dbcollate = defGetString(dcollate);
if (dctype && dctype->arg)
@@ -910,7 +920,9 @@ createdb(ParseState *pstate, const CreatedbStmt *stmt)
{
char *locproviderstr = defGetString(dlocprovider);
- if (pg_strcasecmp(locproviderstr, "icu") == 0)
+ if (pg_strcasecmp(locproviderstr, "builtin") == 0)
+ dblocprovider = COLLPROVIDER_BUILTIN;
+ else if (pg_strcasecmp(locproviderstr, "icu") == 0)
dblocprovider = COLLPROVIDER_ICU;
else if (pg_strcasecmp(locproviderstr, "libc") == 0)
dblocprovider = COLLPROVIDER_LIBC;
@@ -1027,14 +1039,9 @@ createdb(ParseState *pstate, const CreatedbStmt *stmt)
dbctype = src_ctype;
if (dblocprovider == '\0')
dblocprovider = src_locprovider;
- if (dblocale == NULL && dblocprovider == COLLPROVIDER_ICU)
- {
- if (dlocale && dlocale->arg)
- dblocale = defGetString(dlocale);
- else
- dblocale = src_locale;
- }
- if (dbicurules == NULL && dblocprovider == COLLPROVIDER_ICU)
+ if (dblocale == NULL)
+ dblocale = src_locale;
+ if (dbicurules == NULL)
dbicurules = src_icurules;
/* Some encodings are client only */
@@ -1059,6 +1066,27 @@ createdb(ParseState *pstate, const CreatedbStmt *stmt)
check_encoding_locale_matches(encoding, dbcollate, dbctype);
+ if (dblocprovider == COLLPROVIDER_BUILTIN)
+ {
+ /*
+ * This would happen if template0 uses the libc provider but the new
+ * database uses builtin.
+ */
+ if (!dblocale)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("LOCALE must be specified for the builtin provider")));
+
+ dblocale = builtin_validate_locale(encoding, dblocale);
+ }
+ else
+ {
+ if (dbuiltinlocale && dbuiltinlocale->arg)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_OBJECT_DEFINITION),
+ errmsg("BUILTIN_LOCALE cannot be specified unless locale provider is builtin")));
+ }
+
if (dblocprovider == COLLPROVIDER_ICU)
{
if (!(is_encoding_supported_by_icu(encoding)))
@@ -1100,7 +1128,7 @@ createdb(ParseState *pstate, const CreatedbStmt *stmt)
}
else
{
- if (dblocale)
+ if (diculocale && diculocale->arg)
ereport(ERROR,
(errcode(ERRCODE_INVALID_OBJECT_DEFINITION),
errmsg("ICU locale cannot be specified unless locale provider is ICU")));
@@ -1111,6 +1139,10 @@ createdb(ParseState *pstate, const CreatedbStmt *stmt)
errmsg("ICU rules cannot be specified unless locale provider is ICU")));
}
+ /* for libc, locale comes from datcollate and datctype */
+ if (dblocprovider == COLLPROVIDER_LIBC)
+ dblocale = NULL;
+
/*
* Check that the new encoding and locale settings match the source
* database. We insist on this because we simply copy the source data ---
@@ -1195,9 +1227,17 @@ createdb(ParseState *pstate, const CreatedbStmt *stmt)
*/
if (src_collversion && !dcollversion)
{
- char *actual_versionstr;
+ char *actual_versionstr;
+ const char *locale;
- actual_versionstr = get_collation_actual_version(dblocprovider, dblocprovider == COLLPROVIDER_ICU ? dblocale : dbcollate);
+ if (dblocprovider == COLLPROVIDER_ICU)
+ locale = dblocale;
+ else if (dblocprovider == COLLPROVIDER_LIBC)
+ locale = dbcollate;
+ else
+ locale = NULL; /* COLLPROVIDER_BUILTIN */
+
+ actual_versionstr = get_collation_actual_version(dblocprovider, locale);
if (!actual_versionstr)
ereport(ERROR,
(errmsg("template database \"%s\" has a collation version, but no actual collation version could be determined",
@@ -1225,7 +1265,18 @@ createdb(ParseState *pstate, const CreatedbStmt *stmt)
* collation version, which is normally only the case for template0.
*/
if (dbcollversion == NULL)
- dbcollversion = get_collation_actual_version(dblocprovider, dblocprovider == COLLPROVIDER_ICU ? dblocale : dbcollate);
+ {
+ const char *locale;
+
+ if (dblocprovider == COLLPROVIDER_ICU)
+ locale = dblocale;
+ else if (dblocprovider == COLLPROVIDER_LIBC)
+ locale = dbcollate;
+ else
+ locale = NULL; /* COLLPROVIDER_BUILTIN */
+
+ dbcollversion = get_collation_actual_version(dblocprovider, locale);
+ }
/* Resolve default tablespace for new database */
if (dtablespacename && dtablespacename->arg)
@@ -1364,8 +1415,8 @@ createdb(ParseState *pstate, const CreatedbStmt *stmt)
* block on the unique index, and fail after we commit).
*/
- Assert((dblocprovider == COLLPROVIDER_ICU && dblocale) ||
- (dblocprovider != COLLPROVIDER_ICU && !dblocale));
+ Assert((dblocprovider != COLLPROVIDER_LIBC && dblocale) ||
+ (dblocprovider == COLLPROVIDER_LIBC && !dblocale));
/* Form tuple */
new_record[Anum_pg_database_oid - 1] = ObjectIdGetDatum(dboid);
@@ -2446,6 +2497,7 @@ AlterDatabaseRefreshColl(AlterDatabaseRefreshCollStmt *stmt)
ObjectAddress address;
Datum datum;
bool isnull;
+ char *locale;
char *oldversion;
char *newversion;
@@ -2472,10 +2524,24 @@ AlterDatabaseRefreshColl(AlterDatabaseRefreshCollStmt *stmt)
datum = heap_getattr(tuple, Anum_pg_database_datcollversion, RelationGetDescr(rel), &isnull);
oldversion = isnull ? NULL : TextDatumGetCString(datum);
- datum = heap_getattr(tuple, datForm->datlocprovider == COLLPROVIDER_ICU ? Anum_pg_database_datlocale : Anum_pg_database_datcollate, RelationGetDescr(rel), &isnull);
- if (isnull)
- elog(ERROR, "unexpected null in pg_database");
- newversion = get_collation_actual_version(datForm->datlocprovider, TextDatumGetCString(datum));
+ if (datForm->datlocprovider == COLLPROVIDER_ICU)
+ {
+ datum = heap_getattr(tuple, Anum_pg_database_datlocale, RelationGetDescr(rel), &isnull);
+ if (isnull)
+ elog(ERROR, "unexpected null in pg_database");
+ locale = TextDatumGetCString(datum);
+ }
+ else if (datForm->datlocprovider == COLLPROVIDER_LIBC)
+ {
+ datum = heap_getattr(tuple, Anum_pg_database_datcollate, RelationGetDescr(rel), &isnull);
+ if (isnull)
+ elog(ERROR, "unexpected null in pg_database");
+ locale = TextDatumGetCString(datum);
+ }
+ else
+ locale = NULL; /* COLLPROVIDER_BUILTIN */
+
+ newversion = get_collation_actual_version(datForm->datlocprovider, locale);
/* cannot change from NULL to non-NULL or vice versa */
if ((!oldversion && newversion) || (oldversion && !newversion))
@@ -2660,6 +2726,7 @@ pg_database_collation_actual_version(PG_FUNCTION_ARGS)
HeapTuple tp;
char datlocprovider;
Datum datum;
+ char *locale;
char *version;
tp = SearchSysCache1(DATABASEOID, ObjectIdGetDatum(dbid));
@@ -2670,8 +2737,20 @@ pg_database_collation_actual_version(PG_FUNCTION_ARGS)
datlocprovider = ((Form_pg_database) GETSTRUCT(tp))->datlocprovider;
- datum = SysCacheGetAttrNotNull(DATABASEOID, tp, datlocprovider == COLLPROVIDER_ICU ? Anum_pg_database_datlocale : Anum_pg_database_datcollate);
- version = get_collation_actual_version(datlocprovider, TextDatumGetCString(datum));
+ if (datlocprovider == COLLPROVIDER_ICU)
+ {
+ datum = SysCacheGetAttrNotNull(DATABASEOID, tp, Anum_pg_database_datlocale);
+ locale = TextDatumGetCString(datum);
+ }
+ else if (datlocprovider == COLLPROVIDER_LIBC)
+ {
+ datum = SysCacheGetAttrNotNull(DATABASEOID, tp, Anum_pg_database_datcollate);
+ locale = TextDatumGetCString(datum);
+ }
+ else
+ locale = NULL; /* COLLPROVIDER_BUILTIN */
+
+ version = get_collation_actual_version(datlocprovider, locale);
ReleaseSysCache(tp);
diff --git a/src/backend/regex/regc_pg_locale.c b/src/backend/regex/regc_pg_locale.c
index 42d15b6303..0278f45adc 100644
--- a/src/backend/regex/regc_pg_locale.c
+++ b/src/backend/regex/regc_pg_locale.c
@@ -16,6 +16,8 @@
*/
#include "catalog/pg_collation.h"
+#include "common/unicode_case.h"
+#include "common/unicode_category.h"
#include "utils/pg_locale.h"
/*
@@ -64,6 +66,7 @@
typedef enum
{
PG_REGEX_LOCALE_C, /* C locale (encoding independent) */
+ PG_REGEX_BUILTIN, /* built-in Unicode semantics */
PG_REGEX_LOCALE_WIDE, /* Use <wctype.h> functions */
PG_REGEX_LOCALE_1BYTE, /* Use <ctype.h> functions */
PG_REGEX_LOCALE_WIDE_L, /* Use locale_t <wctype.h> functions */
@@ -75,6 +78,8 @@ static PG_Locale_Strategy pg_regex_strategy;
static pg_locale_t pg_regex_locale;
static Oid pg_regex_collation;
+static bool regex_builtin_cclass_posix = false;
+
/*
* Hard-wired character properties for C locale
*/
@@ -266,7 +271,15 @@ pg_set_regex_collation(Oid collation)
if (GetDatabaseEncoding() == PG_UTF8)
{
if (pg_regex_locale)
- pg_regex_strategy = PG_REGEX_LOCALE_WIDE_L;
+ {
+ if (pg_regex_locale->provider == COLLPROVIDER_BUILTIN)
+ {
+ pg_regex_strategy = PG_REGEX_BUILTIN;
+ regex_builtin_cclass_posix = pg_regex_locale->info.builtin.cclass_posix;
+ }
+ else
+ pg_regex_strategy = PG_REGEX_LOCALE_WIDE_L;
+ }
else
pg_regex_strategy = PG_REGEX_LOCALE_WIDE;
}
@@ -290,6 +303,8 @@ pg_wc_isdigit(pg_wchar c)
case PG_REGEX_LOCALE_C:
return (c <= (pg_wchar) 127 &&
(pg_char_properties[c] & PG_ISDIGIT));
+ case PG_REGEX_BUILTIN:
+ return pg_u_isdigit(c, regex_builtin_cclass_posix);
case PG_REGEX_LOCALE_WIDE:
if (sizeof(wchar_t) >= 4 || c <= (pg_wchar) 0xFFFF)
return iswdigit((wint_t) c);
@@ -322,6 +337,8 @@ pg_wc_isalpha(pg_wchar c)
case PG_REGEX_LOCALE_C:
return (c <= (pg_wchar) 127 &&
(pg_char_properties[c] & PG_ISALPHA));
+ case PG_REGEX_BUILTIN:
+ return pg_u_isalpha(c);
case PG_REGEX_LOCALE_WIDE:
if (sizeof(wchar_t) >= 4 || c <= (pg_wchar) 0xFFFF)
return iswalpha((wint_t) c);
@@ -354,6 +371,8 @@ pg_wc_isalnum(pg_wchar c)
case PG_REGEX_LOCALE_C:
return (c <= (pg_wchar) 127 &&
(pg_char_properties[c] & PG_ISALNUM));
+ case PG_REGEX_BUILTIN:
+ return pg_u_isalnum(c, regex_builtin_cclass_posix);
case PG_REGEX_LOCALE_WIDE:
if (sizeof(wchar_t) >= 4 || c <= (pg_wchar) 0xFFFF)
return iswalnum((wint_t) c);
@@ -395,6 +414,8 @@ pg_wc_isupper(pg_wchar c)
case PG_REGEX_LOCALE_C:
return (c <= (pg_wchar) 127 &&
(pg_char_properties[c] & PG_ISUPPER));
+ case PG_REGEX_BUILTIN:
+ return pg_u_isupper(c);
case PG_REGEX_LOCALE_WIDE:
if (sizeof(wchar_t) >= 4 || c <= (pg_wchar) 0xFFFF)
return iswupper((wint_t) c);
@@ -427,6 +448,8 @@ pg_wc_islower(pg_wchar c)
case PG_REGEX_LOCALE_C:
return (c <= (pg_wchar) 127 &&
(pg_char_properties[c] & PG_ISLOWER));
+ case PG_REGEX_BUILTIN:
+ return pg_u_islower(c);
case PG_REGEX_LOCALE_WIDE:
if (sizeof(wchar_t) >= 4 || c <= (pg_wchar) 0xFFFF)
return iswlower((wint_t) c);
@@ -459,6 +482,8 @@ pg_wc_isgraph(pg_wchar c)
case PG_REGEX_LOCALE_C:
return (c <= (pg_wchar) 127 &&
(pg_char_properties[c] & PG_ISGRAPH));
+ case PG_REGEX_BUILTIN:
+ return pg_u_isgraph(c);
case PG_REGEX_LOCALE_WIDE:
if (sizeof(wchar_t) >= 4 || c <= (pg_wchar) 0xFFFF)
return iswgraph((wint_t) c);
@@ -491,6 +516,8 @@ pg_wc_isprint(pg_wchar c)
case PG_REGEX_LOCALE_C:
return (c <= (pg_wchar) 127 &&
(pg_char_properties[c] & PG_ISPRINT));
+ case PG_REGEX_BUILTIN:
+ return pg_u_isprint(c);
case PG_REGEX_LOCALE_WIDE:
if (sizeof(wchar_t) >= 4 || c <= (pg_wchar) 0xFFFF)
return iswprint((wint_t) c);
@@ -523,6 +550,8 @@ pg_wc_ispunct(pg_wchar c)
case PG_REGEX_LOCALE_C:
return (c <= (pg_wchar) 127 &&
(pg_char_properties[c] & PG_ISPUNCT));
+ case PG_REGEX_BUILTIN:
+ return pg_u_ispunct(c, regex_builtin_cclass_posix);
case PG_REGEX_LOCALE_WIDE:
if (sizeof(wchar_t) >= 4 || c <= (pg_wchar) 0xFFFF)
return iswpunct((wint_t) c);
@@ -555,6 +584,8 @@ pg_wc_isspace(pg_wchar c)
case PG_REGEX_LOCALE_C:
return (c <= (pg_wchar) 127 &&
(pg_char_properties[c] & PG_ISSPACE));
+ case PG_REGEX_BUILTIN:
+ return pg_u_isspace(c);
case PG_REGEX_LOCALE_WIDE:
if (sizeof(wchar_t) >= 4 || c <= (pg_wchar) 0xFFFF)
return iswspace((wint_t) c);
@@ -588,6 +619,8 @@ pg_wc_toupper(pg_wchar c)
if (c <= (pg_wchar) 127)
return pg_ascii_toupper((unsigned char) c);
return c;
+ case PG_REGEX_BUILTIN:
+ return unicode_uppercase_simple(c);
case PG_REGEX_LOCALE_WIDE:
/* force C behavior for ASCII characters, per comments above */
if (c <= (pg_wchar) 127)
@@ -628,6 +661,8 @@ pg_wc_tolower(pg_wchar c)
if (c <= (pg_wchar) 127)
return pg_ascii_tolower((unsigned char) c);
return c;
+ case PG_REGEX_BUILTIN:
+ return unicode_lowercase_simple(c);
case PG_REGEX_LOCALE_WIDE:
/* force C behavior for ASCII characters, per comments above */
if (c <= (pg_wchar) 127)
@@ -792,6 +827,9 @@ pg_ctype_get_cache(pg_wc_probefunc probefunc, int cclasscode)
max_chr = (pg_wchar) MAX_SIMPLE_CHR;
#endif
break;
+ case PG_REGEX_BUILTIN:
+ max_chr = (pg_wchar) MAX_SIMPLE_CHR;
+ break;
case PG_REGEX_LOCALE_WIDE:
case PG_REGEX_LOCALE_WIDE_L:
max_chr = (pg_wchar) MAX_SIMPLE_CHR;
@@ -809,6 +847,7 @@ pg_ctype_get_cache(pg_wc_probefunc probefunc, int cclasscode)
max_chr = (pg_wchar) MAX_SIMPLE_CHR;
break;
default:
+ Assert(false);
max_chr = 0; /* can't get here, but keep compiler quiet */
break;
}
diff --git a/src/backend/utils/adt/formatting.c b/src/backend/utils/adt/formatting.c
index d176723d95..c1f04eb03a 100644
--- a/src/backend/utils/adt/formatting.c
+++ b/src/backend/utils/adt/formatting.c
@@ -77,6 +77,8 @@
#include "catalog/pg_collation.h"
#include "catalog/pg_type.h"
+#include "common/unicode_case.h"
+#include "common/unicode_category.h"
#include "mb/pg_wchar.h"
#include "nodes/miscnodes.h"
#include "parser/scansup.h"
@@ -1670,6 +1672,67 @@ str_tolower(const char *buff, size_t nbytes, Oid collid)
}
else
#endif
+ if (mylocale && mylocale->provider == COLLPROVIDER_BUILTIN)
+ {
+ const unsigned char *orig = (unsigned char *) buff;
+ unsigned char *workspace;
+ size_t workspace_size = nbytes + 1;
+ const unsigned char *sp;
+ unsigned char *rp;
+
+ Assert(GetDatabaseEncoding() == PG_UTF8);
+
+ /* Output workspace cannot have more codes than input bytes */
+ workspace = (unsigned char *)palloc(workspace_size);
+
+ sp = orig;
+ rp = workspace;
+ while (sp - orig < nbytes)
+ {
+ pg_wchar u1 = utf8_to_unicode(sp);
+ pg_wchar u2 = unicode_lowercase_simple(u1);
+
+ /*
+ * If we can't fit 4 more bytes, and the next character to
+ * write is multibyte, reallocate buffer to maximum size we
+ * will need.
+ */
+ if (rp - workspace > workspace_size - 4 && u2 >= 0x80)
+ {
+ int written = rp - workspace;
+
+ /* Overflow paranoia */
+ if ((nbytes + 1) > (INT_MAX / sizeof(pg_wchar)))
+ ereport(ERROR,
+ (errcode(ERRCODE_OUT_OF_MEMORY),
+ errmsg("out of memory")));
+
+ workspace_size = (nbytes + 1) * sizeof(pg_wchar);
+ workspace = repalloc(workspace, workspace_size);
+ rp = workspace + written;
+ }
+
+ unicode_to_utf8(u2, rp);
+ sp += pg_utf_mblen(sp);
+ rp += pg_utf_mblen(rp);
+ }
+
+ *rp = '\0';
+ rp++;
+
+ if (workspace_size == rp - workspace)
+ {
+ result = (char *) workspace;
+ }
+ else
+ {
+ /* shrink buffer and store result */
+ result = palloc(rp - workspace);
+ memcpy(result, workspace, rp - workspace);
+ pfree(workspace);
+ }
+ }
+ else
{
if (pg_database_encoding_max_length() > 1)
{
@@ -1788,6 +1851,67 @@ str_toupper(const char *buff, size_t nbytes, Oid collid)
}
else
#endif
+ if (mylocale && mylocale->provider == COLLPROVIDER_BUILTIN)
+ {
+ const unsigned char *orig = (unsigned char *) buff;
+ unsigned char *workspace;
+ size_t workspace_size = nbytes + 1;
+ const unsigned char *sp;
+ unsigned char *rp;
+
+ Assert(GetDatabaseEncoding() == PG_UTF8);
+
+ /* Output workspace cannot have more codes than input bytes */
+ workspace = (unsigned char *)palloc(workspace_size);
+
+ sp = orig;
+ rp = workspace;
+ while (sp - orig < nbytes)
+ {
+ pg_wchar u1 = utf8_to_unicode(sp);
+ pg_wchar u2 = unicode_uppercase_simple(u1);
+
+ /*
+ * If we can't fit 4 more bytes, and the next character to
+ * write is multibyte, reallocate buffer to maximum size we
+ * will need.
+ */
+ if (rp - workspace > workspace_size - 4 && u2 >= 0x80)
+ {
+ int written = rp - workspace;
+
+ /* Overflow paranoia */
+ if ((nbytes + 1) > (INT_MAX / sizeof(pg_wchar)))
+ ereport(ERROR,
+ (errcode(ERRCODE_OUT_OF_MEMORY),
+ errmsg("out of memory")));
+
+ workspace_size = (nbytes + 1) * sizeof(pg_wchar);
+ workspace = repalloc(workspace, workspace_size);
+ rp = workspace + written;
+ }
+
+ unicode_to_utf8(u2, rp);
+ sp += pg_utf_mblen(sp);
+ rp += pg_utf_mblen(rp);
+ }
+
+ *rp = '\0';
+ rp++;
+
+ if (workspace_size == rp - workspace)
+ {
+ result = (char *) workspace;
+ }
+ else
+ {
+ /* shrink buffer and store result */
+ result = palloc(rp - workspace);
+ memcpy(result, workspace, rp - workspace);
+ pfree(workspace);
+ }
+ }
+ else
{
if (pg_database_encoding_max_length() > 1)
{
@@ -1907,6 +2031,75 @@ str_initcap(const char *buff, size_t nbytes, Oid collid)
}
else
#endif
+ if (mylocale && mylocale->provider == COLLPROVIDER_BUILTIN)
+ {
+ const unsigned char *orig = (unsigned char *) buff;
+ unsigned char *workspace;
+ size_t workspace_size = nbytes + 1;
+ const unsigned char *sp;
+ unsigned char *rp;
+
+ Assert(GetDatabaseEncoding() == PG_UTF8);
+
+ /* Output workspace cannot have more codes than input bytes */
+ workspace = (unsigned char *)palloc(workspace_size);
+
+ sp = orig;
+ rp = workspace;
+ while (sp - orig < nbytes)
+ {
+ pg_wchar u1 = utf8_to_unicode(sp);
+ pg_wchar u2;
+
+ if (wasalnum)
+ u2 = unicode_lowercase_simple(u1);
+ else
+ u2 = unicode_titlecase_simple(u1);
+
+ /*
+ * If we can't fit 4 more bytes, and the next character to
+ * write is multibyte, reallocate buffer to maximum size we
+ * will need.
+ */
+ if (rp - workspace > workspace_size - 4 && u2 >= 0x80)
+ {
+ int written = rp - workspace;
+
+ /* Overflow paranoia */
+ if ((nbytes + 1) > (INT_MAX / sizeof(pg_wchar)))
+ ereport(ERROR,
+ (errcode(ERRCODE_OUT_OF_MEMORY),
+ errmsg("out of memory")));
+
+ workspace_size = (nbytes + 1) * sizeof(pg_wchar);
+ workspace = repalloc(workspace, workspace_size);
+ rp = workspace + written;
+ }
+
+ unicode_to_utf8(u2, rp);
+
+ wasalnum = pg_u_isalnum(u2, mylocale->info.builtin.cclass_posix);
+
+ sp += pg_utf_mblen(sp);
+ rp += pg_utf_mblen(rp);
+ }
+
+ *rp = '\0';
+ rp++;
+
+ if (workspace_size == rp - workspace)
+ {
+ result = (char *) workspace;
+ }
+ else
+ {
+ /* shrink buffer and store result */
+ result = palloc(rp - workspace);
+ memcpy(result, workspace, rp - workspace);
+ pfree(workspace);
+ }
+ }
+ else
{
if (pg_database_encoding_max_length() > 1)
{
diff --git a/src/backend/utils/adt/pg_locale.c b/src/backend/utils/adt/pg_locale.c
index 9d8634ec25..7067e29c2d 100644
--- a/src/backend/utils/adt/pg_locale.c
+++ b/src/backend/utils/adt/pg_locale.c
@@ -1269,7 +1269,19 @@ lookup_collation_cache(Oid collation, bool set_flags)
elog(ERROR, "cache lookup failed for collation %u", collation);
collform = (Form_pg_collation) GETSTRUCT(tp);
- if (collform->collprovider == COLLPROVIDER_LIBC)
+ if (collform->collprovider == COLLPROVIDER_BUILTIN)
+ {
+ Datum datum;
+ const char *colllocale;
+
+ datum = SysCacheGetAttrNotNull(COLLOID, tp, Anum_pg_collation_colllocale);
+ colllocale = TextDatumGetCString(datum);
+
+ cache_entry->collate_is_c = true;
+ cache_entry->ctype_is_c = ((strcmp(colllocale, "C") == 0) ||
+ (strcmp(colllocale, "POSIX") == 0));
+ }
+ else if (collform->collprovider == COLLPROVIDER_LIBC)
{
Datum datum;
const char *collcollate;
@@ -1320,16 +1332,30 @@ lc_collate_is_c(Oid collation)
if (collation == DEFAULT_COLLATION_OID)
{
static int result = -1;
- char *localeptr;
-
- if (default_locale.provider == COLLPROVIDER_ICU)
- return false;
+ const char *localeptr;
if (result >= 0)
return (bool) result;
- localeptr = setlocale(LC_COLLATE, NULL);
- if (!localeptr)
- elog(ERROR, "invalid LC_COLLATE setting");
+
+ if (default_locale.provider == COLLPROVIDER_BUILTIN)
+ {
+ result = true;
+ return (bool) result;
+ }
+ else if (default_locale.provider == COLLPROVIDER_ICU)
+ {
+ result = false;
+ return (bool) result;
+ }
+ else if (default_locale.provider == COLLPROVIDER_LIBC)
+ {
+ localeptr = setlocale(LC_CTYPE, NULL);
+ if (!localeptr)
+ elog(ERROR, "invalid LC_CTYPE setting");
+ }
+ else
+ elog(ERROR, "unexpected collation provider '%c'",
+ default_locale.provider);
if (strcmp(localeptr, "C") == 0)
result = true;
@@ -1373,16 +1399,29 @@ lc_ctype_is_c(Oid collation)
if (collation == DEFAULT_COLLATION_OID)
{
static int result = -1;
- char *localeptr;
-
- if (default_locale.provider == COLLPROVIDER_ICU)
- return false;
+ const char *localeptr;
if (result >= 0)
return (bool) result;
- localeptr = setlocale(LC_CTYPE, NULL);
- if (!localeptr)
- elog(ERROR, "invalid LC_CTYPE setting");
+
+ if (default_locale.provider == COLLPROVIDER_BUILTIN)
+ {
+ localeptr = default_locale.info.builtin.locale;
+ }
+ else if (default_locale.provider == COLLPROVIDER_ICU)
+ {
+ result = false;
+ return (bool) result;
+ }
+ else if (default_locale.provider == COLLPROVIDER_LIBC)
+ {
+ localeptr = setlocale(LC_CTYPE, NULL);
+ if (!localeptr)
+ elog(ERROR, "invalid LC_CTYPE setting");
+ }
+ else
+ elog(ERROR, "unexpected collation provider '%c'",
+ default_locale.provider);
if (strcmp(localeptr, "C") == 0)
result = true;
@@ -1390,6 +1429,7 @@ lc_ctype_is_c(Oid collation)
result = true;
else
result = false;
+
return (bool) result;
}
@@ -1520,10 +1560,10 @@ pg_newlocale_from_collation(Oid collid)
if (collid == DEFAULT_COLLATION_OID)
{
- if (default_locale.provider == COLLPROVIDER_ICU)
- return &default_locale;
- else
+ if (default_locale.provider == COLLPROVIDER_LIBC)
return (pg_locale_t) 0;
+ else
+ return &default_locale;
}
cache_entry = lookup_collation_cache(collid, false);
@@ -1548,7 +1588,18 @@ pg_newlocale_from_collation(Oid collid)
result.provider = collform->collprovider;
result.deterministic = collform->collisdeterministic;
- if (collform->collprovider == COLLPROVIDER_LIBC)
+ if (collform->collprovider == COLLPROVIDER_BUILTIN)
+ {
+ const char *locstr;
+
+ datum = SysCacheGetAttrNotNull(COLLOID, tp, Anum_pg_collation_colllocale);
+ locstr = TextDatumGetCString(datum);
+
+ result.info.builtin.locale = MemoryContextStrdup(TopMemoryContext,
+ locstr);
+ result.info.builtin.cclass_posix = true;
+ }
+ else if (collform->collprovider == COLLPROVIDER_LIBC)
{
const char *collcollate;
const char *collctype pg_attribute_unused();
@@ -1627,6 +1678,7 @@ pg_newlocale_from_collation(Oid collid)
collversionstr = TextDatumGetCString(datum);
+ Assert(collform->collprovider != COLLPROVIDER_BUILTIN);
datum = SysCacheGetAttrNotNull(COLLOID, tp, collform->collprovider == COLLPROVIDER_ICU ? Anum_pg_collation_colllocale : Anum_pg_collation_collcollate);
actual_versionstr = get_collation_actual_version(collform->collprovider,
@@ -1678,6 +1730,14 @@ get_collation_actual_version(char collprovider, const char *collcollate)
{
char *collversion = NULL;
+ /*
+ * The only two supported locales (C and C.UTF-8) are both based on memcmp
+ * and do not change. (The ctype behavior can change, but the versioning
+ * does not track that.)
+ */
+ if (collprovider == COLLPROVIDER_BUILTIN)
+ return NULL;
+
#ifdef USE_ICU
if (collprovider == COLLPROVIDER_ICU)
{
@@ -2444,6 +2504,38 @@ pg_strnxfrm_prefix(char *dest, size_t destsize, const char *src,
return result;
}
+const char *
+builtin_validate_locale(int encoding, const char *locale)
+{
+ const char *canonical_name = NULL;
+ int required_encoding = -1;
+
+ if (strcmp(locale, "C") == 0 || strcmp(locale, "POSIX") == 0)
+ {
+ canonical_name = "C";
+ }
+ else if (strcmp(locale, "C.UTF-8") == 0 || strcmp(locale, "C.UTF8") == 0)
+ {
+ required_encoding = PG_UTF8;
+ canonical_name = "C.UTF-8";
+ }
+
+ if (!canonical_name)
+ ereport(ERROR,
+ (errcode(ERRCODE_WRONG_OBJECT_TYPE),
+ errmsg("invalid locale name \"%s\" for builtin provider",
+ locale)));
+
+ if (required_encoding >= 0 && encoding != required_encoding)
+ ereport(ERROR,
+ (errcode(ERRCODE_WRONG_OBJECT_TYPE),
+ errmsg("encoding \"%s\" does not match locale \"%s\"",
+ pg_encoding_to_char(encoding), locale)));
+
+ return canonical_name;
+}
+
+
#ifdef USE_ICU
/*
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 702bad6d8a..f6dc3afbfe 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -318,7 +318,7 @@ CheckMyDatabase(const char *name, bool am_superuser, bool override_allow_connect
bool isnull;
char *collate;
char *ctype;
- char *iculocale;
+ char *datlocale;
/* Fetch our pg_database row normally, via syscache */
tup = SearchSysCache1(DATABASEOID, ObjectIdGetDatum(MyDatabaseId));
@@ -423,12 +423,21 @@ CheckMyDatabase(const char *name, bool am_superuser, bool override_allow_connect
strcmp(ctype, "POSIX") == 0)
database_ctype_is_c = true;
- if (dbform->datlocprovider == COLLPROVIDER_ICU)
+ if (dbform->datlocprovider == COLLPROVIDER_BUILTIN)
+ {
+ datum = SysCacheGetAttrNotNull(DATABASEOID, tup, Anum_pg_database_datlocale);
+ datlocale = TextDatumGetCString(datum);
+
+ default_locale.info.builtin.locale = MemoryContextStrdup(
+ TopMemoryContext, datlocale);
+ default_locale.info.builtin.cclass_posix = true;
+ }
+ else if (dbform->datlocprovider == COLLPROVIDER_ICU)
{
char *icurules;
datum = SysCacheGetAttrNotNull(DATABASEOID, tup, Anum_pg_database_datlocale);
- iculocale = TextDatumGetCString(datum);
+ datlocale = TextDatumGetCString(datum);
datum = SysCacheGetAttr(DATABASEOID, tup, Anum_pg_database_daticurules, &isnull);
if (!isnull)
@@ -436,10 +445,10 @@ CheckMyDatabase(const char *name, bool am_superuser, bool override_allow_connect
else
icurules = NULL;
- make_icu_collator(iculocale, icurules, &default_locale);
+ make_icu_collator(datlocale, icurules, &default_locale);
}
else
- iculocale = NULL;
+ datlocale = NULL;
default_locale.provider = dbform->datlocprovider;
@@ -461,10 +470,16 @@ CheckMyDatabase(const char *name, bool am_superuser, bool override_allow_connect
{
char *actual_versionstr;
char *collversionstr;
+ char *locale;
collversionstr = TextDatumGetCString(datum);
- actual_versionstr = get_collation_actual_version(dbform->datlocprovider, dbform->datlocprovider == COLLPROVIDER_ICU ? iculocale : collate);
+ if (dbform->datlocprovider == COLLPROVIDER_LIBC)
+ locale = collate;
+ else
+ locale = datlocale;
+
+ actual_versionstr = get_collation_actual_version(dbform->datlocprovider, locale);
if (!actual_versionstr)
/* should not happen */
elog(WARNING,
diff --git a/src/bin/initdb/initdb.c b/src/bin/initdb/initdb.c
index eea40b930f..fab9b4c131 100644
--- a/src/bin/initdb/initdb.c
+++ b/src/bin/initdb/initdb.c
@@ -2450,7 +2450,7 @@ usage(const char *progname)
" set default locale in the respective category for\n"
" new databases (default taken from environment)\n"));
printf(_(" --no-locale equivalent to --locale=C\n"));
- printf(_(" --locale-provider={libc|icu}\n"
+ printf(_(" --locale-provider={builtin|libc|icu}\n"
" set default locale provider for new databases\n"));
printf(_(" --pwfile=FILE read password for the new superuser from file\n"));
printf(_(" -T, --text-search-config=CFG\n"
@@ -2601,7 +2601,15 @@ setup_locale_encoding(void)
{
setlocales();
- if (locale_provider == COLLPROVIDER_LIBC &&
+ if (locale_provider == COLLPROVIDER_BUILTIN &&
+ strcmp(lc_ctype, "C") == 0 &&
+ strcmp(lc_collate, "C") == 0 &&
+ strcmp(lc_time, "C") == 0 &&
+ strcmp(lc_numeric, "C") == 0 &&
+ strcmp(lc_monetary, "C") == 0 &&
+ strcmp(lc_messages, "C") == 0)
+ printf(_("The database cluster will be initialized with no locale.\n"));
+ else if (locale_provider == COLLPROVIDER_LIBC &&
strcmp(lc_ctype, lc_collate) == 0 &&
strcmp(lc_ctype, lc_time) == 0 &&
strcmp(lc_ctype, lc_numeric) == 0 &&
@@ -3107,9 +3115,10 @@ main(int argc, char *argv[])
{"allow-group-access", no_argument, NULL, 'g'},
{"discard-caches", no_argument, NULL, 14},
{"locale-provider", required_argument, NULL, 15},
- {"icu-locale", required_argument, NULL, 16},
- {"icu-rules", required_argument, NULL, 17},
- {"sync-method", required_argument, NULL, 18},
+ {"builtin-locale", required_argument, NULL, 16},
+ {"icu-locale", required_argument, NULL, 17},
+ {"icu-rules", required_argument, NULL, 18},
+ {"sync-method", required_argument, NULL, 19},
{NULL, 0, NULL, 0}
};
@@ -3277,7 +3286,9 @@ main(int argc, char *argv[])
"-c debug_discard_caches=1");
break;
case 15:
- if (strcmp(optarg, "icu") == 0)
+ if (strcmp(optarg, "builtin") == 0)
+ locale_provider = COLLPROVIDER_BUILTIN;
+ else if (strcmp(optarg, "icu") == 0)
locale_provider = COLLPROVIDER_ICU;
else if (strcmp(optarg, "libc") == 0)
locale_provider = COLLPROVIDER_LIBC;
@@ -3286,12 +3297,15 @@ main(int argc, char *argv[])
break;
case 16:
datlocale = pg_strdup(optarg);
- icu_locale_specified = true;
break;
case 17:
- icu_rules = pg_strdup(optarg);
+ datlocale = pg_strdup(optarg);
+ icu_locale_specified = true;
break;
case 18:
+ icu_rules = pg_strdup(optarg);
+ break;
+ case 19:
if (!parse_sync_method(optarg, &sync_method))
exit(1);
break;
diff --git a/src/bin/initdb/t/001_initdb.pl b/src/bin/initdb/t/001_initdb.pl
index 7606db1987..1701506abd 100644
--- a/src/bin/initdb/t/001_initdb.pl
+++ b/src/bin/initdb/t/001_initdb.pl
@@ -184,6 +184,61 @@ else
'locale provider ICU fails since no ICU support');
}
+command_fails(
+ [
+ 'initdb', '--no-sync', '--locale-provider=builtin', "$tempdir/data6"
+ ],
+ 'locale provider builtin fails without --locale'
+);
+
+command_ok(
+ [
+ 'initdb', '--no-sync', '--locale-provider=builtin', '--locale=C',
+ "$tempdir/data7"
+ ],
+ 'locale provider builtin with --locale'
+);
+
+command_ok(
+ [
+ 'initdb', '--no-sync', '--locale-provider=builtin', '-E UTF-8',
+ '--builtin-locale=C.UTF-8', "$tempdir/data8"
+ ],
+ 'locale provider builtin with -E UTF-8 --builtin-locale=C.UTF-8'
+);
+
+command_fails(
+ [
+ 'initdb', '--no-sync', '--locale-provider=builtin', '-E SQL_ASCII',
+ '--builtin-locale=C.UTF-8', "$tempdir/data9"
+ ],
+ 'locale provider builtin with --builtin-locale=C.UTF-8 fails for SQL_ASCII'
+);
+
+command_ok(
+ [
+ 'initdb', '--no-sync', '--locale-provider=builtin', '--lc-ctype=C',
+ '--locale=C', "$tempdir/data10"
+ ],
+ 'locale provider builtin with --lc-ctype'
+);
+
+command_fails(
+ [
+ 'initdb', '--no-sync', '--locale-provider=builtin', '--icu-locale=en',
+ "$tempdir/dataX"
+ ],
+ 'fails for locale provider builtin with ICU locale'
+);
+
+command_fails(
+ [
+ 'initdb', '--no-sync', '--locale-provider=builtin', '--icu-rules=""',
+ "$tempdir/dataX"
+ ],
+ 'fails for locale provider builtin with ICU rules'
+);
+
command_fails(
[ 'initdb', '--no-sync', '--locale-provider=xyz', "$tempdir/dataX" ],
'fails for invalid locale provider');
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 3d829c99d7..b953db8bea 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -3091,7 +3091,9 @@ dumpDatabase(Archive *fout)
}
appendPQExpBufferStr(creaQry, " LOCALE_PROVIDER = ");
- if (datlocprovider[0] == 'c')
+ if (datlocprovider[0] == 'b')
+ appendPQExpBufferStr(creaQry, "builtin");
+ else if (datlocprovider[0] == 'c')
appendPQExpBufferStr(creaQry, "libc");
else if (datlocprovider[0] == 'i')
appendPQExpBufferStr(creaQry, "icu");
@@ -13672,7 +13674,9 @@ dumpCollation(Archive *fout, const CollInfo *collinfo)
fmtQualifiedDumpable(collinfo));
appendPQExpBufferStr(q, "provider = ");
- if (collprovider[0] == 'c')
+ if (collprovider[0] == 'b')
+ appendPQExpBufferStr(q, "builtin");
+ else if (collprovider[0] == 'c')
appendPQExpBufferStr(q, "libc");
else if (collprovider[0] == 'i')
appendPQExpBufferStr(q, "icu");
@@ -13693,6 +13697,13 @@ dumpCollation(Archive *fout, const CollInfo *collinfo)
/* no locale -- the default collation cannot be reloaded anyway */
}
+ else if (collprovider[0] == 'b')
+ {
+ if (collcollate || collctype || colllocale || collicurules)
+ pg_log_warning("invalid collation \"%s\"", qcollname);
+
+ appendPQExpBufferStr(q, ", locale = 'C'");
+ }
else if (collprovider[0] == 'i')
{
if (fout->remoteVersion >= 150000)
diff --git a/src/bin/pg_upgrade/t/002_pg_upgrade.pl b/src/bin/pg_upgrade/t/002_pg_upgrade.pl
index 6f359d72ce..37a4e9d334 100644
--- a/src/bin/pg_upgrade/t/002_pg_upgrade.pl
+++ b/src/bin/pg_upgrade/t/002_pg_upgrade.pl
@@ -108,35 +108,71 @@ if ($oldnode->pg_version >= 11)
# can test that pg_upgrade copies the locale settings of template0
# from the old to the new cluster.
-my $original_encoding = "6"; # UTF-8
-my $original_provider = "c";
-my $original_locale = "C";
-my $original_iculocale = "";
-my $provider_field = "'c' AS datlocprovider";
-my $iculocale_field = "NULL AS datlocale";
-if (int($oldnode->pg_version) >= 15 && $ENV{with_icu} eq 'yes')
+my %encoding_number = ('UTF-8' => 6, 'SQL_ASCII' => 0);
+my $provider_field;
+my $datlocale_field;
+my $original_encoding;
+my $original_provider;
+my $original_datcollate = "C";
+my $original_datctype = "C";
+my $original_datlocale;
+
+if (int($oldnode->pg_version) >= 15)
{
$provider_field = "datlocprovider";
if (int($oldnode->pg_version) >= 17)
{
- $iculocale_field = "datlocale";
+ $datlocale_field = "datlocale";
}
else
{
- $iculocale_field = "daticulocale AS datlocale";
+ $datlocale_field = "daticulocale AS datlocale";
}
+}
+else
+{
+ $provider_field = "'c' AS datlocprovider";
+ $datlocale_field = "NULL AS datlocale";
+}
+
+if (int($oldnode->pg_version) >= 17)
+{
+ $original_encoding = "UTF-8";
+ $original_provider = "b";
+ $original_datlocale = "C.UTF-8";
+}
+elsif (int($oldnode->pg_version) >= 15 && $ENV{with_icu} eq 'yes')
+{
+ $original_encoding = "UTF-8";
$original_provider = "i";
- $original_iculocale = "fr-CA";
+ $original_datlocale = "fr-CA";
+}
+else
+{
+ my $original_encoding = "SQL_ASCII";
+ my $original_provider = "c";
+ my $original_datlocale = "";
}
my @initdb_params = @custom_opts;
-push @initdb_params, ('--encoding', 'UTF-8');
-push @initdb_params, ('--locale', $original_locale);
-if ($original_provider eq "i")
+push @initdb_params, ('--encoding', $original_encoding);
+push @initdb_params, ('--lc-collate', $original_datcollate);
+push @initdb_params, ('--lc-ctype', $original_datctype);
+
+# add --locale-provider, if supported
+my %provider_name = ('b' => 'builtin', 'i' => 'icu', 'c' => 'libc');
+if ($oldnode->pg_version >= 15)
{
- push @initdb_params, ('--locale-provider', 'icu');
- push @initdb_params, ('--icu-locale', 'fr-CA');
+ push @initdb_params, ('--locale-provider', $provider_name{$original_provider});
+ if ($original_provider eq 'b')
+ {
+ push @initdb_params, ('--builtin-locale', $original_datlocale);
+ }
+ elsif ($original_provider eq 'i')
+ {
+ push @initdb_params, ('--icu-locale', $original_datlocale);
+ }
}
$node_params{extra} = \@initdb_params;
@@ -146,10 +182,10 @@ $oldnode->start;
my $result;
$result = $oldnode->safe_psql(
'postgres',
- "SELECT encoding, $provider_field, datcollate, datctype, $iculocale_field
+ "SELECT encoding, $provider_field, datcollate, datctype, $datlocale_field
FROM pg_database WHERE datname='template0'");
is( $result,
- "$original_encoding|$original_provider|$original_locale|$original_locale|$original_iculocale",
+ "$encoding_number{$original_encoding}|$original_provider|$original_datcollate|$original_datctype|$original_datlocale",
"check locales in original cluster");
# The default location of the source code is the root of this directory.
@@ -429,10 +465,10 @@ if (-d $log_path)
# Test that upgraded cluster has original locale settings.
$result = $newnode->safe_psql(
'postgres',
- "SELECT encoding, $provider_field, datcollate, datctype, $iculocale_field
+ "SELECT encoding, $provider_field, datcollate, datctype, $datlocale_field
FROM pg_database WHERE datname='template0'");
is( $result,
- "$original_encoding|$original_provider|$original_locale|$original_locale|$original_iculocale",
+ "$encoding_number{$original_encoding}|$original_provider|$original_datcollate|$original_datctype|$original_datlocale",
"check that locales in new cluster match original cluster");
# Second dump from the upgraded instance.
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index 1122843715..3b327d159a 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -926,7 +926,7 @@ listAllDbs(const char *pattern, bool verbose)
gettext_noop("Encoding"));
if (pset.sversion >= 150000)
appendPQExpBuffer(&buf,
- " CASE d.datlocprovider WHEN 'c' THEN 'libc' WHEN 'i' THEN 'icu' END AS \"%s\",\n",
+ " CASE d.datlocprovider WHEN 'b' THEN 'builtin' WHEN 'c' THEN 'libc' WHEN 'i' THEN 'icu' END AS \"%s\",\n",
gettext_noop("Locale Provider"));
else
appendPQExpBuffer(&buf,
@@ -4966,7 +4966,7 @@ listCollations(const char *pattern, bool verbose, bool showSystem)
if (pset.sversion >= 100000)
appendPQExpBuffer(&buf,
- " CASE c.collprovider WHEN 'd' THEN 'default' WHEN 'c' THEN 'libc' WHEN 'i' THEN 'icu' END AS \"%s\",\n",
+ " CASE c.collprovider WHEN 'd' THEN 'default' WHEN 'b' THEN 'builtin' WHEN 'c' THEN 'libc' WHEN 'i' THEN 'icu' END AS \"%s\",\n",
gettext_noop("Provider"));
else
appendPQExpBuffer(&buf,
diff --git a/src/bin/scripts/createdb.c b/src/bin/scripts/createdb.c
index 9ca86a3e53..78d729106a 100644
--- a/src/bin/scripts/createdb.c
+++ b/src/bin/scripts/createdb.c
@@ -40,8 +40,9 @@ main(int argc, char *argv[])
{"locale", required_argument, NULL, 'l'},
{"maintenance-db", required_argument, NULL, 3},
{"locale-provider", required_argument, NULL, 4},
- {"icu-locale", required_argument, NULL, 5},
- {"icu-rules", required_argument, NULL, 6},
+ {"builtin-locale", required_argument, NULL, 5},
+ {"icu-locale", required_argument, NULL, 6},
+ {"icu-rules", required_argument, NULL, 7},
{NULL, 0, NULL, 0}
};
@@ -67,6 +68,7 @@ main(int argc, char *argv[])
char *lc_ctype = NULL;
char *locale = NULL;
char *locale_provider = NULL;
+ char *builtin_locale = NULL;
char *icu_locale = NULL;
char *icu_rules = NULL;
@@ -134,9 +136,12 @@ main(int argc, char *argv[])
locale_provider = pg_strdup(optarg);
break;
case 5:
- icu_locale = pg_strdup(optarg);
+ builtin_locale = pg_strdup(optarg);
break;
case 6:
+ icu_locale = pg_strdup(optarg);
+ break;
+ case 7:
icu_rules = pg_strdup(optarg);
break;
default:
@@ -216,6 +221,11 @@ main(int argc, char *argv[])
appendPQExpBufferStr(&sql, " LOCALE ");
appendStringLiteralConn(&sql, locale, conn);
}
+ if (builtin_locale)
+ {
+ appendPQExpBufferStr(&sql, " BUILTIN_LOCALE ");
+ appendStringLiteralConn(&sql, builtin_locale, conn);
+ }
if (lc_collate)
{
appendPQExpBufferStr(&sql, " LC_COLLATE ");
@@ -296,7 +306,7 @@ help(const char *progname)
printf(_(" --lc-ctype=LOCALE LC_CTYPE setting for the database\n"));
printf(_(" --icu-locale=LOCALE ICU locale setting for the database\n"));
printf(_(" --icu-rules=RULES ICU rules setting for the database\n"));
- printf(_(" --locale-provider={libc|icu}\n"
+ printf(_(" --locale-provider={builtin|libc|icu}\n"
" locale provider for the database's default collation\n"));
printf(_(" -O, --owner=OWNER database user to own the new database\n"));
printf(_(" -S, --strategy=STRATEGY database creation strategy wal_log or file_copy\n"));
diff --git a/src/bin/scripts/t/020_createdb.pl b/src/bin/scripts/t/020_createdb.pl
index 40291924e5..feee9cf85d 100644
--- a/src/bin/scripts/t/020_createdb.pl
+++ b/src/bin/scripts/t/020_createdb.pl
@@ -105,6 +105,78 @@ else
'create database with ICU fails since no ICU support');
}
+$node->command_fails(
+ [
+ 'createdb', '-T', 'template0', '--locale-provider=builtin',
+ 'tbuiltin1'
+ ],
+ 'create database with provider "builtin" fails without --locale'
+);
+
+$node->command_ok(
+ [
+ 'createdb', '-T', 'template0', '--locale-provider=builtin',
+ '--locale=C', 'tbuiltin2'
+ ],
+ 'create database with provider "builtin" and locale "C"'
+);
+
+$node->command_ok(
+ [
+ 'createdb', '-T', 'template0', '--locale-provider=builtin',
+ '--locale=C', '--lc-collate=C', 'tbuiltin3'
+ ],
+ 'create database with provider "builtin" and LC_COLLATE=C'
+);
+
+$node->command_ok(
+ [
+ 'createdb', '-T', 'template0', '--locale-provider=builtin',
+ '--locale=C', '--lc-ctype=C', 'tbuiltin4'
+ ],
+ 'create database with provider "builtin" and LC_CTYPE=C'
+);
+
+$node->command_ok(
+ [
+ 'createdb', '-T', 'template0', '--locale-provider=builtin',
+ '-E UTF-8', '--builtin-locale=C.UTF8', 'tbuiltin5'
+ ],
+ 'create database with provider "builtin" and --builtin-locale C.UTF-8'
+);
+
+$node->command_fails(
+ [
+ 'createdb', '-T', 'template0', '--locale-provider=builtin',
+ '-E LATIN1', '--builtin-locale=C.UTF-8', 'tbuiltin6'
+ ],
+ 'create database with provider "builtin" and --builtin-locale C.UTF-8'
+);
+
+$node->command_fails(
+ [
+ 'createdb', '-T', 'template0', '--locale-provider=builtin',
+ '--locale=C', '--icu-locale=en', 'tbuiltin7'
+ ],
+ 'create database with provider "builtin" and ICU_LOCALE="en"'
+);
+
+$node->command_fails(
+ [
+ 'createdb', '-T', 'template0', '--locale-provider=builtin',
+ '--locale=C', '--icu-rules=""', 'tbuiltin8'
+ ],
+ 'create database with provider "builtin" and ICU_RULES=""'
+);
+
+$node->command_fails(
+ [
+ 'createdb', '-T', 'template1', '--locale-provider=builtin',
+ '--locale=C', 'tbuiltin9'
+ ],
+ 'create database with provider "builtin" not matching template'
+);
+
$node->command_fails([ 'createdb', 'foobar1' ],
'fails if database already exists');
diff --git a/src/include/catalog/pg_collation.dat b/src/include/catalog/pg_collation.dat
index 2c112cd6bc..fa071ce68a 100644
--- a/src/include/catalog/pg_collation.dat
+++ b/src/include/catalog/pg_collation.dat
@@ -23,12 +23,15 @@
descr => 'standard POSIX collation',
collname => 'POSIX', collprovider => 'c', collencoding => '-1',
collcollate => 'POSIX', collctype => 'POSIX' },
-{ oid => '962', descr => 'sorts by Unicode code point',
- collname => 'ucs_basic', collprovider => 'c', collencoding => '6',
- collcollate => 'C', collctype => 'C' },
+{ oid => '962', descr => 'sorts by Unicode code point, C character semantics',
+ collname => 'ucs_basic', collprovider => 'b', collencoding => '6',
+ colllocale => 'C' },
{ oid => '963',
descr => 'sorts using the Unicode Collation Algorithm with default settings',
collname => 'unicode', collprovider => 'i', collencoding => '-1',
colllocale => 'und' },
+{ oid => '970', descr => 'sorts by Unicode code point; Unicode & POSIX character semantics',
+ collname => 'c_utf8', collprovider => 'b', collencoding => '6',
+ colllocale => 'C.UTF8' },
]
diff --git a/src/include/catalog/pg_collation.h b/src/include/catalog/pg_collation.h
index d357c89ae6..09fc991038 100644
--- a/src/include/catalog/pg_collation.h
+++ b/src/include/catalog/pg_collation.h
@@ -65,6 +65,7 @@ DECLARE_UNIQUE_INDEX_PKEY(pg_collation_oid_index, 3085, CollationOidIndexId, pg_
#ifdef EXPOSE_TO_CLIENT_CODE
#define COLLPROVIDER_DEFAULT 'd'
+#define COLLPROVIDER_BUILTIN 'b'
#define COLLPROVIDER_ICU 'i'
#define COLLPROVIDER_LIBC 'c'
@@ -73,6 +74,8 @@ collprovider_name(char c)
{
switch (c)
{
+ case COLLPROVIDER_BUILTIN:
+ return "builtin";
case COLLPROVIDER_ICU:
return "icu";
case COLLPROVIDER_LIBC:
diff --git a/src/include/utils/pg_locale.h b/src/include/utils/pg_locale.h
index 6447bea8e0..22dc38f42e 100644
--- a/src/include/utils/pg_locale.h
+++ b/src/include/utils/pg_locale.h
@@ -76,6 +76,11 @@ struct pg_locale_struct
bool deterministic;
union
{
+ struct
+ {
+ const char *locale;
+ bool cclass_posix;
+ } builtin;
locale_t lt;
#ifdef USE_ICU
struct
@@ -112,7 +117,7 @@ extern size_t pg_strxfrm_prefix(char *dest, const char *src, size_t destsize,
pg_locale_t locale);
extern size_t pg_strnxfrm_prefix(char *dest, size_t destsize, const char *src,
size_t srclen, pg_locale_t locale);
-
+extern const char *builtin_validate_locale(int encoding, const char *loc_str);
extern void icu_validate_locale(const char *loc_str);
extern char *icu_language_tag(const char *loc_str, int elevel);
diff --git a/src/test/icu/t/010_database.pl b/src/test/icu/t/010_database.pl
index 67fc3bbf19..af34cfc1dd 100644
--- a/src/test/icu/t/010_database.pl
+++ b/src/test/icu/t/010_database.pl
@@ -63,14 +63,14 @@ is( $node1->psql(
0,
"C locale works for ICU");
-# Test that LOCALE works for ICU locales if LC_COLLATE and LC_CTYPE
-# are specified
-is( $node1->psql(
- 'postgres',
- q{CREATE DATABASE dbicu2 LOCALE_PROVIDER icu LOCALE '@colStrength=primary'
- LC_COLLATE='C' LC_CTYPE='C' TEMPLATE template0 ENCODING UTF8}
- ),
- 0,
- "LOCALE works for ICU locales if LC_COLLATE and LC_CTYPE are specified");
+my ($ret, $stdout, $stderr) = $node1->psql('postgres',
+ q{CREATE DATABASE dbicu LOCALE_PROVIDER builtin LOCALE 'C' TEMPLATE dbicu}
+);
+isnt($ret, 0,
+ "locale provider must match template: exit code not 0");
+like(
+ $stderr,
+ qr/ERROR: new locale provider \(builtin\) does not match locale provider of the template database \(icu\)/,
+ "locale provider must match template: error message");
done_testing();
diff --git a/src/test/regress/expected/collate.out b/src/test/regress/expected/collate.out
index 0649564485..ece4a8e99d 100644
--- a/src/test/regress/expected/collate.out
+++ b/src/test/regress/expected/collate.out
@@ -650,6 +650,26 @@ EXPLAIN (COSTS OFF)
(3 rows)
-- CREATE/DROP COLLATION
+CREATE COLLATION builtin_c ( PROVIDER = builtin, LOCALE = "C" );
+CREATE COLLATION builtin_posix ( PROVIDER = builtin, LOCALE = "POSIX" );
+SELECT b FROM collate_test1 ORDER BY b COLLATE builtin_c;
+ b
+-----
+ ABD
+ Abc
+ abc
+ bbc
+(4 rows)
+
+CREATE COLLATION builtin2 ( PROVIDER = builtin ); -- fails
+ERROR: parameter "locale" must be specified
+CREATE COLLATION builtin2 ( PROVIDER = builtin, LOCALE = "en_US" ); -- fails
+ERROR: invalid locale name "en_US" for builtin provider
+CREATE COLLATION builtin2 ( PROVIDER = builtin, LC_CTYPE = "C", LC_COLLATE = "C" ); -- fails
+ERROR: parameter "locale" must be specified
+CREATE COLLATION builtin2 ( PROVIDER = builtin, LOCALE = "POSIX", LC_CTYPE = "POSIX" ); -- fails
+ERROR: conflicting or redundant options
+DETAIL: LOCALE cannot be specified together with LC_COLLATE or LC_CTYPE.
CREATE COLLATION mycoll1 FROM "C";
CREATE COLLATION mycoll2 ( LC_COLLATE = "POSIX", LC_CTYPE = "POSIX" );
CREATE COLLATION mycoll3 FROM "default"; -- intentionally unsupported
@@ -754,7 +774,7 @@ DETAIL: FROM cannot be specified together with any other options.
-- must get rid of them.
--
DROP SCHEMA collate_tests CASCADE;
-NOTICE: drop cascades to 19 other objects
+NOTICE: drop cascades to 21 other objects
DETAIL: drop cascades to table collate_test1
drop cascades to table collate_test_like
drop cascades to table collate_test2
@@ -771,6 +791,8 @@ drop cascades to function dup(anyelement)
drop cascades to table collate_test20
drop cascades to table collate_test21
drop cascades to table collate_test22
+drop cascades to collation builtin_c
+drop cascades to collation builtin_posix
drop cascades to collation mycoll2
drop cascades to table collate_test23
drop cascades to view collate_on_int
diff --git a/src/test/regress/expected/collate.utf8.out b/src/test/regress/expected/collate.utf8.out
new file mode 100644
index 0000000000..f4a9d26978
--- /dev/null
+++ b/src/test/regress/expected/collate.utf8.out
@@ -0,0 +1,97 @@
+/*
+ * This test is for collations and character operations when using the
+ * builtin provider with the C.UTF-8 locale.
+ */
+/* skip test if not UTF8 server encoding */
+SELECT getdatabaseencoding() <> 'UTF8' AS skip_test \gset
+\if :skip_test
+\quit
+\endif
+SET client_encoding TO UTF8;
+--
+-- Test preinstalled C_UTF8 collation.
+--
+CREATE TABLE builtin_test (
+ t TEXT COLLATE C_UTF8
+);
+INSERT INTO builtin_test VALUES
+ ('abc DEF'),
+ ('ábc DÉF'),
+ ('DŽxxDŽ džxxDž Džxxdž');
+SELECT t, lower(t), initcap(t), upper(t) FROM builtin_test;
+ t | lower | initcap | upper
+----------------+----------------+----------------+----------------
+ abc DEF | abc def | Abc Def | ABC DEF
+ ábc DÉF | ábc déf | Ábc Déf | ÁBC DÉF
+ DŽxxDŽ džxxDž Džxxdž | džxxdž džxxdž džxxdž | Džxxdž Džxxdž Džxxdž | DŽXXDŽ DŽXXDŽ DŽXXDŽ
+(3 rows)
+
+DROP TABLE builtin_test;
+-- character classes
+SELECT 'xyz' ~ '[[:alnum:]]' COLLATE C_UTF8;
+ ?column?
+----------
+ t
+(1 row)
+
+SELECT 'xyz' !~ '[[:upper:]]' COLLATE C_UTF8;
+ ?column?
+----------
+ t
+(1 row)
+
+SELECT '@' !~ '[[:alnum:]]' COLLATE C_UTF8;
+ ?column?
+----------
+ t
+(1 row)
+
+SELECT '@' ~ '[[:punct:]]' COLLATE C_UTF8; -- symbols are punctuation in posix
+ ?column?
+----------
+ t
+(1 row)
+
+SELECT 'a8a' ~ '[[:digit:]]' COLLATE C_UTF8;
+ ?column?
+----------
+ t
+(1 row)
+
+SELECT '൧' !~ '\d' COLLATE C_UTF8; -- only 0-9 considered digits in posix
+ ?column?
+----------
+ t
+(1 row)
+
+-- case mapping
+SELECT 'xYz' ~* 'XyZ' COLLATE C_UTF8;
+ ?column?
+----------
+ t
+(1 row)
+
+SELECT 'xAb' ~* '[W-Y]' COLLATE C_UTF8;
+ ?column?
+----------
+ t
+(1 row)
+
+SELECT 'xAb' !~* '[c-d]' COLLATE C_UTF8;
+ ?column?
+----------
+ t
+(1 row)
+
+SELECT 'Δ' ~* '[α-λ]' COLLATE C_UTF8;
+ ?column?
+----------
+ t
+(1 row)
+
+SELECT 'δ' ~* '[Γ-Λ]' COLLATE C_UTF8; -- same as above with cases reversed
+ ?column?
+----------
+ t
+(1 row)
+
diff --git a/src/test/regress/expected/collate.utf8_1.out b/src/test/regress/expected/collate.utf8_1.out
new file mode 100644
index 0000000000..e73fdf50c3
--- /dev/null
+++ b/src/test/regress/expected/collate.utf8_1.out
@@ -0,0 +1,8 @@
+/*
+ * This test is for collations and character operations when using the
+ * builtin provider with the C.UTF-8 locale.
+ */
+/* skip test if not UTF8 server encoding */
+SELECT getdatabaseencoding() <> 'UTF8' AS skip_test \gset
+\if :skip_test
+\quit
diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule
index f0987ff537..292bc54932 100644
--- a/src/test/regress/parallel_schedule
+++ b/src/test/regress/parallel_schedule
@@ -78,9 +78,9 @@ test: brin_bloom brin_multi
# psql depends on create_am
# amutils depends on geometry, create_index_spgist, hash_index, brin
# ----------
-test: create_table_like alter_generic alter_operator misc async dbsize merge misc_functions sysviews tsrf tid tidscan tidrangescan collate.icu.utf8 incremental_sort create_role
+test: create_table_like alter_generic alter_operator misc async dbsize merge misc_functions sysviews tsrf tid tidscan tidrangescan collate.utf8 collate.icu.utf8 incremental_sort create_role
-# collate.*.utf8 tests cannot be run in parallel with each other
+# collate.linux.utf8 and collate.icu.utf8 tests cannot be run in parallel with each other
test: rules psql psql_crosstab amutils stats_ext collate.linux.utf8 collate.windows.win1252
# ----------
diff --git a/src/test/regress/sql/collate.sql b/src/test/regress/sql/collate.sql
index c3d40fc195..01d5c69fe4 100644
--- a/src/test/regress/sql/collate.sql
+++ b/src/test/regress/sql/collate.sql
@@ -244,6 +244,16 @@ EXPLAIN (COSTS OFF)
-- CREATE/DROP COLLATION
+CREATE COLLATION builtin_c ( PROVIDER = builtin, LOCALE = "C" );
+CREATE COLLATION builtin_posix ( PROVIDER = builtin, LOCALE = "POSIX" );
+
+SELECT b FROM collate_test1 ORDER BY b COLLATE builtin_c;
+
+CREATE COLLATION builtin2 ( PROVIDER = builtin ); -- fails
+CREATE COLLATION builtin2 ( PROVIDER = builtin, LOCALE = "en_US" ); -- fails
+CREATE COLLATION builtin2 ( PROVIDER = builtin, LC_CTYPE = "C", LC_COLLATE = "C" ); -- fails
+CREATE COLLATION builtin2 ( PROVIDER = builtin, LOCALE = "POSIX", LC_CTYPE = "POSIX" ); -- fails
+
CREATE COLLATION mycoll1 FROM "C";
CREATE COLLATION mycoll2 ( LC_COLLATE = "POSIX", LC_CTYPE = "POSIX" );
CREATE COLLATION mycoll3 FROM "default"; -- intentionally unsupported
diff --git a/src/test/regress/sql/collate.utf8.sql b/src/test/regress/sql/collate.utf8.sql
new file mode 100644
index 0000000000..0f30c5704d
--- /dev/null
+++ b/src/test/regress/sql/collate.utf8.sql
@@ -0,0 +1,45 @@
+/*
+ * This test is for collations and character operations when using the
+ * builtin provider with the C.UTF-8 locale.
+ */
+
+/* skip test if not UTF8 server encoding */
+SELECT getdatabaseencoding() <> 'UTF8' AS skip_test \gset
+\if :skip_test
+\quit
+\endif
+
+SET client_encoding TO UTF8;
+
+--
+-- Test preinstalled C_UTF8 collation.
+--
+
+CREATE TABLE builtin_test (
+ t TEXT COLLATE C_UTF8
+);
+INSERT INTO builtin_test VALUES
+ ('abc DEF'),
+ ('ábc DÉF'),
+ ('DŽxxDŽ džxxDž Džxxdž');
+
+SELECT t, lower(t), initcap(t), upper(t) FROM builtin_test;
+
+DROP TABLE builtin_test;
+
+-- character classes
+
+SELECT 'xyz' ~ '[[:alnum:]]' COLLATE C_UTF8;
+SELECT 'xyz' !~ '[[:upper:]]' COLLATE C_UTF8;
+SELECT '@' !~ '[[:alnum:]]' COLLATE C_UTF8;
+SELECT '@' ~ '[[:punct:]]' COLLATE C_UTF8; -- symbols are punctuation in posix
+SELECT 'a8a' ~ '[[:digit:]]' COLLATE C_UTF8;
+SELECT '൧' !~ '\d' COLLATE C_UTF8; -- only 0-9 considered digits in posix
+
+-- case mapping
+
+SELECT 'xYz' ~* 'XyZ' COLLATE C_UTF8;
+SELECT 'xAb' ~* '[W-Y]' COLLATE C_UTF8;
+SELECT 'xAb' !~* '[c-d]' COLLATE C_UTF8;
+SELECT 'Δ' ~* '[α-λ]' COLLATE C_UTF8;
+SELECT 'δ' ~* '[Γ-Λ]' COLLATE C_UTF8; -- same as above with cases reversed
--
2.34.1
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: Built-in CTYPE provider
2023-12-29 02:57 Re: Built-in CTYPE provider Jeff Davis <[email protected]>
@ 2024-01-09 01:17 ` Jeremy Schneider <[email protected]>
2024-01-09 22:55 ` Re: Built-in CTYPE provider Jeff Davis <[email protected]>
2 siblings, 1 reply; 9+ messages in thread
From: Jeremy Schneider @ 2024-01-09 01:17 UTC (permalink / raw)
To: Jeff Davis <[email protected]>; Daniel Verite <[email protected]>; +Cc: Robert Haas <[email protected]>; pgsql-hackers
On 12/28/23 6:57 PM, Jeff Davis wrote:
>
> Attached a more complete version that fixes a few bugs, stabilizes the
> tests, and improves the documentation. I optimized the performance, too
> -- now it's beating both libc's "C.utf8" and ICU "en-US-x-icu" for both
> collation and case mapping (numbers below).
>
> It's really nice to finally be able to have platform-independent tests
> that work on any UTF-8 database.
Thanks for all your work on this, Jeff
I didn't know about the Unicode stability policy. Since it's formal
policy, I agree this provides some assumptions we can safely build on.
I'm working my way through these patches but it's taking a little time
for me. I hadn't tracked with the "builtin" thread last summer so I'm
coming up to speed on that now too. I'm hopeful that something along
these lines gets into pg17. The pg17 cycle is going to start heating up
pretty soon.
I agree with merging the threads, even though it makes for a larger
patch set. It would be great to get a unified "builtin" provider in
place for the next major.
I also still want to parse my way through your email reply about the two
groups of callers, and what this means for user experience.
https://www.postgresql.org/message-id/7774b3a64f51b3375060c29871cf2b02b3e85dab.camel%40j-davis.com
> Let's separate it into groups.
> (1) Callers that use a collation OID or pg_locale_t:
> (2) A long tail of callers that depend on what LC_CTYPE/LC_COLLATE are
> set to, or use ad-hoc ASCII-only semantics:
In the first list it seems that some callers might be influenced by a
COLLATE clause or table definition while others always take the database
default? It still seems a bit odd to me if different providers can be
used for different parts of a single SQL. But it might not be so bad - I
haven't fully thought through it yet and I'm still kicking the tires on
my test build over here.
Is there any reason we couldn't commit the minor cleanup (patch 0001)
now? It's less than 200 lines and pretty straightforward.
I wonder if, after a year of running the builtin provider in production,
whether we might consider adding to the builtin provider a few locales
with simple but more reasonable ordering for european and asian
languages? Maybe just grabbing a current version of DUCET with no
intention of ever updating it? I don't know how bad sorting is with
plain DUCET for things like french or spanish or german, but surely it's
not as unusable as code point order? Anyone who needs truly accurate or
updated or customized linguistic sorting can always go to ICU, and take
responsibility for their ICU upgrades, but something basic and static
might meet the needs of 99% of postgres users indefinitely.
By the way - my coworker Josh (who I don't think posts much on the
hackers list here, but shares an unhealthy inability to look away from
database unicode disasters) passed along this link today which I think
is a fantastic list of surprises about programming and strings
(generally unicode).
https://jeremyhussell.blogspot.com/2017/11/falsehoods-programmers-believe-about.html#main
Make sure to click the link to show the counterexamples and discussion,
that's the best part.
-Jeremy
PS. I was joking around today that the the second best part is that it's
proof that people named Jeremy are always brilliant within their field.
😂 Josh said its just a subset of "always trust people whose names start
with J" which seems fair. Unfortunately I can't yet think of a way to
shoehorn the rest of the amazing PG hackers on this thread into the joke.
--
http://about.me/jeremy_schneider
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: Built-in CTYPE provider
2023-12-29 02:57 Re: Built-in CTYPE provider Jeff Davis <[email protected]>
2024-01-09 01:17 ` Re: Built-in CTYPE provider Jeremy Schneider <[email protected]>
@ 2024-01-09 22:55 ` Jeff Davis <[email protected]>
0 siblings, 0 replies; 9+ messages in thread
From: Jeff Davis @ 2024-01-09 22:55 UTC (permalink / raw)
To: Jeremy Schneider <[email protected]>; Daniel Verite <[email protected]>; +Cc: Robert Haas <[email protected]>; pgsql-hackers
On Mon, 2024-01-08 at 17:17 -0800, Jeremy Schneider wrote:
> I agree with merging the threads, even though it makes for a larger
> patch set. It would be great to get a unified "builtin" provider in
> place for the next major.
I believe that's possible and that this proposal is quite close (hoping
to get something in this 'fest). The tables I'm introducing have
exhaustive test coverage, so there's not a lot of risk there. And the
builtin provider itself is an optional feature, so it won't be
disruptive.
>
> In the first list it seems that some callers might be influenced by a
> COLLATE clause or table definition while others always take the
> database
> default? It still seems a bit odd to me if different providers can be
> used for different parts of a single SQL.
Right, that can happen today, and my proposal doesn't change that.
Basically those are cases where the caller was never properly onboarded
to our collation system, like the ts_locale.c routines.
> Is there any reason we couldn't commit the minor cleanup (patch 0001)
> now? It's less than 200 lines and pretty straightforward.
Sure, I'll commit that fairly soon then.
> I wonder if, after a year of running the builtin provider in
> production,
> whether we might consider adding to the builtin provider a few
> locales
> with simple but more reasonable ordering for european and asian
> languages?
I won't rule that out completely, but there's a lot we would need to do
to get there. Even assuming we implement that perfectly, we'd need to
make sure it's a reasonable scope for Postgres as a project and that we
have more than one person willing to maintain it. Similar things have
been rejected before for similar reasons.
What I'm proposing for v17 is much simpler: basically some lookup
tables, which is just an extension of what we're already doing for
normalization.
> https://jeremyhussell.blogspot.com/2017/11/falsehoods-programmers-believe-about.html#main
>
> Make sure to click the link to show the counterexamples and
> discussion,
> that's the best part.
Yes, it can be hard to reason about this stuff but I believe Unicode
provides a lot of good data and guidance to work from. If you think my
proposal relies on one of those assumptions let me know.
To the extent that I do rely on any of those assumptions, it's mostly
to match libc's "C.UTF-8" behavior.
Regards,
Jeff Davis
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: Built-in CTYPE provider
2023-12-29 02:57 Re: Built-in CTYPE provider Jeff Davis <[email protected]>
@ 2024-01-09 22:17 ` Jeremy Schneider <[email protected]>
2024-01-09 22:31 ` Re: Built-in CTYPE provider Jeff Davis <[email protected]>
2024-01-11 23:36 ` Re: Built-in CTYPE provider Jeff Davis <[email protected]>
2 siblings, 2 replies; 9+ messages in thread
From: Jeremy Schneider @ 2024-01-09 22:17 UTC (permalink / raw)
To: Jeff Davis <[email protected]>; Daniel Verite <[email protected]>; +Cc: Robert Haas <[email protected]>; pgsql-hackers
On 12/28/23 6:57 PM, Jeff Davis wrote:
> On Wed, 2023-12-27 at 17:26 -0800, Jeff Davis wrote:
> Attached a more complete version that fixes a few bugs, stabilizes the
> tests, and improves the documentation. I optimized the performance, too
> -- now it's beating both libc's "C.utf8" and ICU "en-US-x-icu" for both
> collation and case mapping (numbers below).
>
> It's really nice to finally be able to have platform-independent tests
> that work on any UTF-8 database.
I think we missed something in psql, pretty sure I applied all the
patches but I see this error:
=# \l
ERROR: 42703: column d.datlocale does not exist
LINE 8: d.datlocale as "Locale",
^
HINT: Perhaps you meant to reference the column "d.daticulocale".
LOCATION: errorMissingColumn, parse_relation.c:3720
=====
This is interesting. Jeff your original email didn't explicitly show any
other initcap() results, but on Ubuntu 22.04 (glibc 2.35) I see
different results:
=# SELECT initcap('axxE áxxÉ DŽxxDŽ Džxxx džxxx');
initcap
--------------------------
Axxe Áxxé DŽxxdž DŽxxx DŽxxx
=# SELECT initcap('axxE áxxÉ DŽxxDŽ Džxxx džxxx' COLLATE C_UTF8);
initcap
--------------------------
Axxe Áxxé Džxxdž Džxxx Džxxx
The COLLATE sql syntax feels awkward to me. In this example, we're just
using it to attach locale info to the string, and there's not actually
any collation involved here. Not sure if COLLATE comes from the
standard, and even if it does I'm not sure whether the standard had
upper/lowercase in mind.
That said, I think the thing that mainly matters will be the CREATE
DATABASE syntax and the database default.
I want to try a few things with table-level defaults that differ from
database-level defaults, especially table-level ICU defaults because I
think a number of PostgreSQL users set that up in the years before we
supported DB-level ICU. Some people will probably keep using their
old/existing schema-creation scripts even after they begin provisioning
new systems with new database-level defaults.
-Jeremy
--
http://about.me/jeremy_schneider
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: Built-in CTYPE provider
2023-12-29 02:57 Re: Built-in CTYPE provider Jeff Davis <[email protected]>
2024-01-09 22:17 ` Re: Built-in CTYPE provider Jeremy Schneider <[email protected]>
@ 2024-01-09 22:31 ` Jeff Davis <[email protected]>
2024-01-10 07:35 ` Re: Built-in CTYPE provider Jeremy Schneider <[email protected]>
1 sibling, 1 reply; 9+ messages in thread
From: Jeff Davis @ 2024-01-09 22:31 UTC (permalink / raw)
To: Jeremy Schneider <[email protected]>; Daniel Verite <[email protected]>; +Cc: Robert Haas <[email protected]>; pgsql-hackers
On Tue, 2024-01-09 at 14:17 -0800, Jeremy Schneider wrote:
> I think we missed something in psql, pretty sure I applied all the
> patches but I see this error:
>
> =# \l
> ERROR: 42703: column d.datlocale does not exist
> LINE 8: d.datlocale as "Locale",
>
Thank you. I'll fix this in the next patch set.
> This is interesting. Jeff your original email didn't explicitly show
> any
> other initcap() results, but on Ubuntu 22.04 (glibc 2.35) I see
> different results:
>
> =# SELECT initcap('axxE áxxÉ DŽxxDŽ Džxxx džxxx');
> initcap
> --------------------------
> Axxe Áxxé DŽxxdž DŽxxx DŽxxx
>
> =# SELECT initcap('axxE áxxÉ DŽxxDŽ Džxxx džxxx' COLLATE C_UTF8);
> initcap
> --------------------------
> Axxe Áxxé Džxxdž Džxxx Džxxx
The reason for this difference is because libc doesn't support
titlecase. In the next patch set, I'll not use titlecase (only
uppercase/lowercase even for initcap()), and then it will match libc
100%.
> The COLLATE sql syntax feels awkward to me. In this example, we're
> just
> using it to attach locale info to the string, and there's not
> actually
> any collation involved here. Not sure if COLLATE comes from the
> standard, and even if it does I'm not sure whether the standard had
> upper/lowercase in mind.
The standard doesn't use the COLLATE clause for case mapping, but also
doesn't offer any other mechanism to, e.g., get case mapping according
to the "tr_TR" locale.
I think what Postgres does here, re-purposing the COLLATE clause to
allow tailoring of case mapping, is imperfect but reasonable. My
proposal doesn't change that.
Regards,
Jeff Davis
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: Built-in CTYPE provider
2023-12-29 02:57 Re: Built-in CTYPE provider Jeff Davis <[email protected]>
2024-01-09 22:17 ` Re: Built-in CTYPE provider Jeremy Schneider <[email protected]>
2024-01-09 22:31 ` Re: Built-in CTYPE provider Jeff Davis <[email protected]>
@ 2024-01-10 07:35 ` Jeremy Schneider <[email protected]>
0 siblings, 0 replies; 9+ messages in thread
From: Jeremy Schneider @ 2024-01-10 07:35 UTC (permalink / raw)
To: Jeff Davis <[email protected]>; +Cc: pgsql-hackers
On 1/9/24 2:31 PM, Jeff Davis wrote:
> On Tue, 2024-01-09 at 14:17 -0800, Jeremy Schneider wrote:
>> I think we missed something in psql, pretty sure I applied all the
>> patches but I see this error:
>>
>> =# \l
>> ERROR: 42703: column d.datlocale does not exist
>> LINE 8: d.datlocale as "Locale",
>>
>
> Thank you. I'll fix this in the next patch set.
Very minor nitpick/request. Not directly with this patch set but with
the category_test which is related and also recently committed.
I'm testing out "make update-unicode" scripts, and due to my system ICU
version being different from the PostgreSQL unicode version I get the
expected warnings from category_test:
Postgres Unicode Version: 15.1
ICU Unicode Version: 14.0
Skipped 5116 codepoints unassigned in ICU due to Unicode version mismatch.
category_test: All tests successful!
I know it's minor, but I saw the 5116 skipped codepoints and saw "all
tests succeeded" but I thought the output might be a little nicer if we
showed the count of tests that succeeded. For example:
Postgres Unicode Version: 15.1
ICU Unicode Version: 14.0
Skipped 5116 codepoints unassigned in ICU due to Unicode version mismatch.
category_test: All 1108996 tests successful!
It's a minor tweak to a script that I don't think even runs in the
standard build; any objections? Patch attached.
-Jeremy
--
http://about.me/jeremy_schneider
From 5ff09eb7371abc14cd8537b4e3265e35f030794f Mon Sep 17 00:00:00 2001
From: Jeremy Schneider <[email protected]>
Date: Wed, 10 Jan 2024 07:25:17 +0000
Subject: [PATCH] Output count of checked codepoints.
---
src/common/unicode/category_test.c | 8 +++++++-
1 file changed, 7 insertions(+), 1 deletion(-)
diff --git a/src/common/unicode/category_test.c b/src/common/unicode/category_test.c
index 6cd7cd1a5f..62ff345066 100644
--- a/src/common/unicode/category_test.c
+++ b/src/common/unicode/category_test.c
@@ -53,6 +53,7 @@ main(int argc, char **argv)
int icu_unicode_version = parse_unicode_version(U_UNICODE_VERSION);
int pg_skipped_codepoints = 0;
int icu_skipped_codepoints = 0;
+ int checked_codepoints = 0;
printf("Postgres Unicode Version:\t%s\n", PG_UNICODE_VERSION);
printf("ICU Unicode Version:\t\t%s\n", U_UNICODE_VERSION);
@@ -90,6 +91,10 @@ main(int argc, char **argv)
exit(1);
}
}
+ else
+ {
+ checked_codepoints++;
+ }
}
if (pg_skipped_codepoints > 0)
@@ -99,7 +104,8 @@ main(int argc, char **argv)
printf("Skipped %d codepoints unassigned in ICU due to Unicode version mismatch.\n",
icu_skipped_codepoints);
- printf("category_test: All tests successful!\n");
+ printf("category_test: All %d tests successful!\n",
+ checked_codepoints);
exit(0);
#else
printf("ICU support required for test; skipping.\n");
--
2.34.1
Attachments:
[text/plain] 0001-Output-count-of-checked-codepoints.patch (1.4K, ../../[email protected]/2-0001-Output-count-of-checked-codepoints.patch)
download | inline diff:
From 5ff09eb7371abc14cd8537b4e3265e35f030794f Mon Sep 17 00:00:00 2001
From: Jeremy Schneider <[email protected]>
Date: Wed, 10 Jan 2024 07:25:17 +0000
Subject: [PATCH] Output count of checked codepoints.
---
src/common/unicode/category_test.c | 8 +++++++-
1 file changed, 7 insertions(+), 1 deletion(-)
diff --git a/src/common/unicode/category_test.c b/src/common/unicode/category_test.c
index 6cd7cd1a5f..62ff345066 100644
--- a/src/common/unicode/category_test.c
+++ b/src/common/unicode/category_test.c
@@ -53,6 +53,7 @@ main(int argc, char **argv)
int icu_unicode_version = parse_unicode_version(U_UNICODE_VERSION);
int pg_skipped_codepoints = 0;
int icu_skipped_codepoints = 0;
+ int checked_codepoints = 0;
printf("Postgres Unicode Version:\t%s\n", PG_UNICODE_VERSION);
printf("ICU Unicode Version:\t\t%s\n", U_UNICODE_VERSION);
@@ -90,6 +91,10 @@ main(int argc, char **argv)
exit(1);
}
}
+ else
+ {
+ checked_codepoints++;
+ }
}
if (pg_skipped_codepoints > 0)
@@ -99,7 +104,8 @@ main(int argc, char **argv)
printf("Skipped %d codepoints unassigned in ICU due to Unicode version mismatch.\n",
icu_skipped_codepoints);
- printf("category_test: All tests successful!\n");
+ printf("category_test: All %d tests successful!\n",
+ checked_codepoints);
exit(0);
#else
printf("ICU support required for test; skipping.\n");
--
2.34.1
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: Built-in CTYPE provider
2023-12-29 02:57 Re: Built-in CTYPE provider Jeff Davis <[email protected]>
2024-01-09 22:17 ` Re: Built-in CTYPE provider Jeremy Schneider <[email protected]>
@ 2024-01-11 23:36 ` Jeff Davis <[email protected]>
1 sibling, 0 replies; 9+ messages in thread
From: Jeff Davis @ 2024-01-11 23:36 UTC (permalink / raw)
To: Jeremy Schneider <[email protected]>; Daniel Verite <[email protected]>; +Cc: Robert Haas <[email protected]>; pgsql-hackers
On Tue, 2024-01-09 at 14:17 -0800, Jeremy Schneider wrote:
> I think we missed something in psql, pretty sure I applied all the
> patches but I see this error:
>
> =# \l
> ERROR: 42703: column d.datlocale does not exist
> LINE 8: d.datlocale as "Locale",
> ^
> HINT: Perhaps you meant to reference the column "d.daticulocale".
> LOCATION: errorMissingColumn, parse_relation.c:3720
I think you're connecting to a patched server with an older version of
psql, so it doesn't know the catalog column was renamed.
pg_dump and pg_upgrade don't have that problem because they throw an
error when connecting to a newer server.
But for psql, that's perfectly reasonable to connect to a newer server.
Have we dropped or renamed catalog columns used by psql backslash
commands before, and if so, how do we handle that?
I can just not rename that column, but that's a bit frustrating because
it means I'd need to add a new column to pg_database, which seems
redundant.
Regards,
Jeff Davis
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: Built-in CTYPE provider
2023-12-29 02:57 Re: Built-in CTYPE provider Jeff Davis <[email protected]>
@ 2024-01-10 22:56 ` Daniel Verite <[email protected]>
2 siblings, 0 replies; 9+ messages in thread
From: Daniel Verite @ 2024-01-10 22:56 UTC (permalink / raw)
To: Jeff Davis <[email protected]>; +Cc: Robert Haas <[email protected]>; Jeremy Schneider <[email protected]>; pgsql-hackers
Jeff Davis wrote:
> Attached a more complete version that fixes a few bugs
[v15 patch]
When selecting the builtin provider with initdb, I'm getting the
following setup:
$ bin/initdb --locale=C.UTF-8 --locale-provider=builtin -D/tmp/pgdata
The database cluster will be initialized with this locale configuration:
default collation provider: builtin
default collation locale: C.UTF-8
LC_COLLATE: C.UTF-8
LC_CTYPE: C.UTF-8
LC_MESSAGES: C.UTF-8
LC_MONETARY: C.UTF-8
LC_NUMERIC: C.UTF-8
LC_TIME: C.UTF-8
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".
This is from an environment where LANG=fr_FR.UTF-8
I would expect all LC_* variables to be fr_FR.UTF-8, and the default
text search configuration to be "french". It is what happens
when selecting ICU as the provider in the same environment:
$ bin/initdb --icu-locale=en --locale-provider=icu -D/tmp/pgdata
Using language tag "en" for ICU locale "en".
The database cluster will be initialized with this locale configuration:
default collation provider: icu
default collation locale: en
LC_COLLATE: fr_FR.UTF-8
LC_CTYPE: fr_FR.UTF-8
LC_MESSAGES: fr_FR.UTF-8
LC_MONETARY: fr_FR.UTF-8
LC_NUMERIC: fr_FR.UTF-8
LC_TIME: fr_FR.UTF-8
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "french".
The collation setup does not influence the rest of the localization.
The problem AFAIU is that --locale has two distinct
meanings in the v15 patch:
--locale-provider=X --locale=Y means use "X" as the provider
with "Y" as datlocale, and it means use "Y" as the locale for all
localized libc functionalities.
I wonder what would happen if invoking
bin/initdb --locale=C.UTF-8 --locale-provider=builtin -D/tmp/pgdata
on a system where C.UTF-8 does not exist as a libc locale.
Would it fail? (I don't have an OS like this to test ATM, will try later).
A related comment is about naming the builtin locale C.UTF-8, the same
name as in libc. On one hand this is semantically sound, but on the
other hand, it's likely to confuse people. What about using completely
different names, like "pg_unicode" or something else prefixed by "pg_"
both for the locale name and the collation name (currently
C.UTF-8/c_utf8)?
Best regards,
--
Daniel Vérité
https://postgresql.verite.pro/
Twitter: @DanielVerite
^ permalink raw reply [nested|flat] 9+ messages in thread
* [PATCH] Limit PL/Perl scalar copies to work_mem
@ 2026-07-06 22:13 Andrey Rachitskiy <[email protected]>
0 siblings, 0 replies; 9+ messages in thread
From: Andrey Rachitskiy @ 2026-07-06 22:13 UTC (permalink / raw)
When a PL/Perl function returns a very large text value, sv2cstr() copies
the entire Perl string into backend memory with no size check. A user
with permission to create untrusted PL/Perl functions can return strings
far larger than work_mem and risk getting the backend killed by the OOM
killer.
Reject Perl strings larger than work_mem * 1024 bytes, capped by
MaxAllocSize, before copying them through sv2cstr(). This follows the
same work_mem-based limit pattern used elsewhere in the backend.
Add a plperl regression test that attempts to return a 16MB string with
the default 4MB work_mem setting.
Author: Andrey Rachitskiy <[email protected]>
---
src/pl/plperl/expected/plperl.out | 8 +++++++
src/pl/plperl/plperl.h | 34 ++++++++++++++++++++++++++++++
src/pl/plperl/sql/plperl.sql | 7 ++++++
3 files changed, 49 insertions(+)
diff --git a/src/pl/plperl/expected/plperl.out b/src/pl/plperl/expected/plperl.out
index e3d7c88..50c788b 100644
--- a/src/pl/plperl/expected/plperl.out
+++ b/src/pl/plperl/expected/plperl.out
@@ -792,3 +792,11 @@ SELECT self_modify(42);
126
(1 row)
+-- oversized text results are rejected at the PL boundary
+CREATE OR REPLACE FUNCTION perl_oversized_text() RETURNS text AS $$
+ return 'x' x (16 * 1024 * 1024);
+$$ LANGUAGE plperl;
+SELECT perl_oversized_text();
+ERROR: Perl value exceeds maximum allowed size (4194304 bytes)
+HINT: Increase work_mem or reduce the result size.
+CONTEXT: PL/Perl function "perl_oversized_text"
diff --git a/src/pl/plperl/plperl.h b/src/pl/plperl/plperl.h
index 4c03f9e..1ccce6d 100644
--- a/src/pl/plperl/plperl.h
+++ b/src/pl/plperl/plperl.h
@@ -17,6 +17,8 @@
/* defines free() by way of system headers, so must be included before perl.h */
#include "mb/pg_wchar.h"
+#include "miscadmin.h"
+#include "utils/memutils.h"
/*
* Pull in Perl headers via a wrapper header, to control the scope of
@@ -40,6 +42,36 @@ char *plperl_sv_to_literal(SV *, char *);
void plperl_util_elog(int level, SV *msg);
+/*
+ * Maximum byte size for a Perl scalar copied through sv2cstr().
+ *
+ * This follows the same work_mem * 1024 pattern used elsewhere in the
+ * backend (e.g. reorderbuffer.c, nodeHash.c) and is capped by MaxAllocSize.
+ */
+static inline Size
+plperl_max_scalar_bytes(void)
+{
+ Size limit = (Size) work_mem * (Size) 1024;
+
+ return Min(limit, MaxAllocSize - 1);
+}
+
+/*
+ * Reject Perl strings that are too large to copy into backend memory.
+ */
+static inline void
+plperl_check_sv_length(STRLEN len)
+{
+ Size max_len = plperl_max_scalar_bytes();
+
+ if ((Size) len > max_len)
+ erereport(ERROR,
+ (errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED),
+ errmsg("Perl value exceeds maximum allowed size (%zu bytes)",
+ max_len),
+ errhint("Increase work_mem or reduce the result size.")));
+}
+
/* helper functions */
/*
@@ -127,6 +159,8 @@ sv2cstr(SV *sv)
else
val = SvPVutf8(sv, len);
+ plperl_check_sv_length(len);
+
/*
* Now convert to database encoding. We use perl's length in the event we
* had an embedded null byte to ensure we error out properly.
diff --git a/src/pl/plperl/sql/plperl.sql b/src/pl/plperl/sql/plperl.sql
index bb0b8ce..0470a2b 100644
--- a/src/pl/plperl/sql/plperl.sql
+++ b/src/pl/plperl/sql/plperl.sql
@@ -521,3 +521,10 @@ $$ LANGUAGE plperl;
SELECT self_modify(42);
SELECT self_modify(42);
+
+-- oversized text results are rejected at the PL boundary
+CREATE OR REPLACE FUNCTION perl_oversized_text() RETURNS text AS $$
+ return 'x' x (16 * 1024 * 1024);
+$$ LANGUAGE plperl;
+
+SELECT perl_oversized_text();
--MP_/sDTAhto+gkG/PMs8Alw6m8e--
^ permalink raw reply [nested|flat] 9+ messages in thread
end of thread, other threads:[~2026-07-06 22:13 UTC | newest]
Thread overview: 9+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2023-12-29 02:57 Re: Built-in CTYPE provider Jeff Davis <[email protected]>
2024-01-09 01:17 ` Jeremy Schneider <[email protected]>
2024-01-09 22:55 ` Jeff Davis <[email protected]>
2024-01-09 22:17 ` Jeremy Schneider <[email protected]>
2024-01-09 22:31 ` Jeff Davis <[email protected]>
2024-01-10 07:35 ` Jeremy Schneider <[email protected]>
2024-01-11 23:36 ` Jeff Davis <[email protected]>
2024-01-10 22:56 ` Daniel Verite <[email protected]>
2026-07-06 22:13 [PATCH] Limit PL/Perl scalar copies to work_mem Andrey Rachitskiy <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox