public inbox for [email protected]help / color / mirror / Atom feed
[PATCH v4 1/2] Add 'noError' argument to encoding conversion functions. 3+ messages / 3 participants [nested] [flat]
* [PATCH v4 1/2] Add 'noError' argument to encoding conversion functions. @ 2021-02-07 15:10 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 3+ messages in thread From: Heikki Linnakangas @ 2021-02-07 15:10 UTC (permalink / raw) With the 'noError' argument, you can try to convert a buffer without knowing the character boundaries beforehand. The functions now need to return the number of input bytes successfully converted. This is is a backwards-incompatible change, if you have created a custom encoding conversion with CREATE CONVERSION. This adds a check to pg_upgrade for that, refusing the upgrade if there are any user-defined encoding conversions. Add regression tests for built-in encoding conversions. This doesn't cover every conversion, but it covers all the internal functions in conv.c that are used to implement the conversions. --- doc/src/sgml/ref/create_conversion.sgml | 5 +- src/backend/commands/conversioncmds.c | 30 +- src/backend/utils/error/elog.c | 2 + src/backend/utils/mb/conv.c | 139 ++++- .../cyrillic_and_mic/cyrillic_and_mic.c | 127 +++-- .../euc2004_sjis2004/euc2004_sjis2004.c | 94 +++- .../euc_cn_and_mic/euc_cn_and_mic.c | 57 +- .../euc_jp_and_sjis/euc_jp_and_sjis.c | 153 ++++-- .../euc_kr_and_mic/euc_kr_and_mic.c | 57 +- .../euc_tw_and_big5/euc_tw_and_big5.c | 165 ++++-- .../latin2_and_win1250/latin2_and_win1250.c | 49 +- .../latin_and_mic/latin_and_mic.c | 43 +- .../utf8_and_big5/utf8_and_big5.c | 37 +- .../utf8_and_cyrillic/utf8_and_cyrillic.c | 67 ++- .../utf8_and_euc2004/utf8_and_euc2004.c | 37 +- .../utf8_and_euc_cn/utf8_and_euc_cn.c | 37 +- .../utf8_and_euc_jp/utf8_and_euc_jp.c | 37 +- .../utf8_and_euc_kr/utf8_and_euc_kr.c | 37 +- .../utf8_and_euc_tw/utf8_and_euc_tw.c | 37 +- .../utf8_and_gb18030/utf8_and_gb18030.c | 37 +- .../utf8_and_gbk/utf8_and_gbk.c | 37 +- .../utf8_and_iso8859/utf8_and_iso8859.c | 43 +- .../utf8_and_iso8859_1/utf8_and_iso8859_1.c | 35 +- .../utf8_and_johab/utf8_and_johab.c | 37 +- .../utf8_and_sjis/utf8_and_sjis.c | 37 +- .../utf8_and_sjis2004/utf8_and_sjis2004.c | 37 +- .../utf8_and_uhc/utf8_and_uhc.c | 37 +- .../utf8_and_win/utf8_and_win.c | 43 +- src/backend/utils/mb/mbutils.c | 76 ++- src/bin/pg_upgrade/check.c | 95 ++++ src/include/catalog/pg_proc.dat | 332 +++++------ src/include/mb/pg_wchar.h | 35 +- src/test/regress/expected/conversion.out | 519 ++++++++++++++++++ src/test/regress/expected/opr_sanity.out | 7 +- .../regress/input/create_function_1.source | 4 + .../regress/output/create_function_1.source | 3 + src/test/regress/regress.c | 134 +++++ src/test/regress/sql/conversion.sql | 185 +++++++ src/test/regress/sql/opr_sanity.sql | 7 +- 39 files changed, 2322 insertions(+), 628 deletions(-) diff --git a/doc/src/sgml/ref/create_conversion.sgml b/doc/src/sgml/ref/create_conversion.sgml index e7700fecfc5..f014a676c88 100644 --- a/doc/src/sgml/ref/create_conversion.sgml +++ b/doc/src/sgml/ref/create_conversion.sgml @@ -117,8 +117,9 @@ conv_proc( integer, -- destination encoding ID cstring, -- source string (null terminated C string) internal, -- destination (fill with a null terminated C string) - integer -- source string length -) RETURNS void; + integer, -- source string length + boolean -- if true, don't throw an error if conversion fails +) RETURNS integer; </programlisting></para> </listitem> </varlistentry> diff --git a/src/backend/commands/conversioncmds.c b/src/backend/commands/conversioncmds.c index f7ff321de71..59e73000206 100644 --- a/src/backend/commands/conversioncmds.c +++ b/src/backend/commands/conversioncmds.c @@ -45,8 +45,9 @@ CreateConversionCommand(CreateConversionStmt *stmt) const char *from_encoding_name = stmt->for_encoding_name; const char *to_encoding_name = stmt->to_encoding_name; List *func_name = stmt->func_name; - static const Oid funcargs[] = {INT4OID, INT4OID, CSTRINGOID, INTERNALOID, INT4OID}; + static const Oid funcargs[] = {INT4OID, INT4OID, CSTRINGOID, INTERNALOID, INT4OID, BOOLOID}; char result[1]; + Datum funcresult; /* Convert list of names to a name and namespace */ namespaceId = QualifiedNameGetCreationNamespace(stmt->conversion_name, @@ -92,8 +93,8 @@ CreateConversionCommand(CreateConversionStmt *stmt) funcoid = LookupFuncName(func_name, sizeof(funcargs) / sizeof(Oid), funcargs, false); - /* Check it returns VOID, else it's probably the wrong function */ - if (get_func_rettype(funcoid) != VOIDOID) + /* Check it returns int4, else it's probably the wrong function */ + if (get_func_rettype(funcoid) != INT4OID) ereport(ERROR, (errcode(ERRCODE_INVALID_OBJECT_DEFINITION), errmsg("encoding conversion function %s must return type %s", @@ -111,12 +112,23 @@ CreateConversionCommand(CreateConversionStmt *stmt) * string; the conversion function should throw an error if it can't * perform the requested conversion. */ - OidFunctionCall5(funcoid, - Int32GetDatum(from_encoding), - Int32GetDatum(to_encoding), - CStringGetDatum(""), - CStringGetDatum(result), - Int32GetDatum(0)); + funcresult = OidFunctionCall6(funcoid, + Int32GetDatum(from_encoding), + Int32GetDatum(to_encoding), + CStringGetDatum(""), + CStringGetDatum(result), + Int32GetDatum(0), + BoolGetDatum(false)); + + /* + * The function should return 0 for empty input. Might as well check that, + * too. + */ + if (DatumGetInt32(funcresult) != 0) + ereport(ERROR, + (errcode(ERRCODE_INVALID_OBJECT_DEFINITION), + errmsg("encoding conversion function %s returned incorrect result for empty input", + NameListToString(func_name)))); /* * All seem ok, go ahead (possible failure would be a duplicate conversion diff --git a/src/backend/utils/error/elog.c b/src/backend/utils/error/elog.c index 80c26724612..762f77d533c 100644 --- a/src/backend/utils/error/elog.c +++ b/src/backend/utils/error/elog.c @@ -2280,6 +2280,8 @@ write_console(const char *line, int len) * Conversion on non-win32 platforms is not implemented yet. It requires * non-throw version of pg_do_encoding_conversion(), that converts * unconvertable characters to '?' without errors. + * + * XXX: We have a no-throw version now. It doesn't convert to '?' though. */ #endif diff --git a/src/backend/utils/mb/conv.c b/src/backend/utils/mb/conv.c index a07b54bd3b8..33e9c9a9e3c 100644 --- a/src/backend/utils/mb/conv.c +++ b/src/backend/utils/mb/conv.c @@ -25,15 +25,20 @@ * tab holds conversion entries for the source charset * starting from 128 (0x80). each entry in the table holds the corresponding * code point for the target charset, or 0 if there is no equivalent code. + * + * Returns the number of input bytes consumed. If noError is true, this can + * be less than 'len'. */ -void +int local2local(const unsigned char *l, unsigned char *p, int len, int src_encoding, int dest_encoding, - const unsigned char *tab) + const unsigned char *tab, + bool noError) { + const unsigned char *start = l; unsigned char c1, c2; @@ -41,7 +46,11 @@ local2local(const unsigned char *l, { c1 = *l; if (c1 == 0) + { + if (noError) + break; report_invalid_encoding(src_encoding, (const char *) l, len); + } if (!IS_HIGHBIT_SET(c1)) *p++ = c1; else @@ -50,13 +59,19 @@ local2local(const unsigned char *l, if (c2) *p++ = c2; else + { + if (noError) + break; report_untranslatable_char(src_encoding, dest_encoding, (const char *) l, len); + } } l++; len--; } *p = '\0'; + + return l - start; } /* @@ -66,18 +81,26 @@ local2local(const unsigned char *l, * p is the output area (must be large enough!) * lc is the mule character set id for the local encoding * encoding is the PG identifier for the local encoding + * + * Returns the number of input bytes consumed. If noError is true, this can + * be less than 'len'. */ -void +int latin2mic(const unsigned char *l, unsigned char *p, int len, - int lc, int encoding) + int lc, int encoding, bool noError) { + const unsigned char *start = l; int c1; while (len > 0) { c1 = *l; if (c1 == 0) + { + if (noError) + break; report_invalid_encoding(encoding, (const char *) l, len); + } if (IS_HIGHBIT_SET(c1)) *p++ = lc; *p++ = c1; @@ -85,6 +108,8 @@ latin2mic(const unsigned char *l, unsigned char *p, int len, len--; } *p = '\0'; + + return l - start; } /* @@ -94,18 +119,26 @@ latin2mic(const unsigned char *l, unsigned char *p, int len, * p is the output area (must be large enough!) * lc is the mule character set id for the local encoding * encoding is the PG identifier for the local encoding + * + * Returns the number of input bytes consumed. If noError is true, this can + * be less than 'len'. */ -void +int mic2latin(const unsigned char *mic, unsigned char *p, int len, - int lc, int encoding) + int lc, int encoding, bool noError) { + const unsigned char *start = mic; int c1; while (len > 0) { c1 = *mic; if (c1 == 0) + { + if (noError) + break; report_invalid_encoding(PG_MULE_INTERNAL, (const char *) mic, len); + } if (!IS_HIGHBIT_SET(c1)) { /* easy for ASCII */ @@ -118,17 +151,27 @@ mic2latin(const unsigned char *mic, unsigned char *p, int len, int l = pg_mule_mblen(mic); if (len < l) + { + if (noError) + break; report_invalid_encoding(PG_MULE_INTERNAL, (const char *) mic, len); + } if (l != 2 || c1 != lc || !IS_HIGHBIT_SET(mic[1])) + { + if (noError) + break; report_untranslatable_char(PG_MULE_INTERNAL, encoding, (const char *) mic, len); + } *p++ = mic[1]; mic += 2; len -= 2; } } *p = '\0'; + + return mic - start; } @@ -143,15 +186,20 @@ mic2latin(const unsigned char *mic, unsigned char *p, int len, * tab holds conversion entries for the local charset * starting from 128 (0x80). each entry in the table holds the corresponding * code point for the mule encoding, or 0 if there is no equivalent code. + * + * Returns the number of input bytes consumed. If noError is true, this can + * be less than 'len'. */ -void +int latin2mic_with_table(const unsigned char *l, unsigned char *p, int len, int lc, int encoding, - const unsigned char *tab) + const unsigned char *tab, + bool noError) { + const unsigned char *start = l; unsigned char c1, c2; @@ -159,7 +207,11 @@ latin2mic_with_table(const unsigned char *l, { c1 = *l; if (c1 == 0) + { + if (noError) + break; report_invalid_encoding(encoding, (const char *) l, len); + } if (!IS_HIGHBIT_SET(c1)) *p++ = c1; else @@ -171,13 +223,19 @@ latin2mic_with_table(const unsigned char *l, *p++ = c2; } else + { + if (noError) + break; report_untranslatable_char(encoding, PG_MULE_INTERNAL, (const char *) l, len); + } } l++; len--; } *p = '\0'; + + return l - start; } /* @@ -191,15 +249,20 @@ latin2mic_with_table(const unsigned char *l, * tab holds conversion entries for the mule internal code's second byte, * starting from 128 (0x80). each entry in the table holds the corresponding * code point for the local charset, or 0 if there is no equivalent code. + * + * Returns the number of input bytes consumed. If noError is true, this can + * be less than 'len'. */ -void +int mic2latin_with_table(const unsigned char *mic, unsigned char *p, int len, int lc, int encoding, - const unsigned char *tab) + const unsigned char *tab, + bool noError) { + const unsigned char *start = mic; unsigned char c1, c2; @@ -207,7 +270,11 @@ mic2latin_with_table(const unsigned char *mic, { c1 = *mic; if (c1 == 0) + { + if (noError) + break; report_invalid_encoding(PG_MULE_INTERNAL, (const char *) mic, len); + } if (!IS_HIGHBIT_SET(c1)) { /* easy for ASCII */ @@ -220,11 +287,17 @@ mic2latin_with_table(const unsigned char *mic, int l = pg_mule_mblen(mic); if (len < l) + { + if (noError) + break; report_invalid_encoding(PG_MULE_INTERNAL, (const char *) mic, len); + } if (l != 2 || c1 != lc || !IS_HIGHBIT_SET(mic[1]) || (c2 = tab[mic[1] - HIGHBIT]) == 0) { + if (noError) + break; report_untranslatable_char(PG_MULE_INTERNAL, encoding, (const char *) mic, len); break; /* keep compiler quiet */ @@ -235,6 +308,8 @@ mic2latin_with_table(const unsigned char *mic, } } *p = '\0'; + + return mic - start; } /* @@ -424,18 +499,22 @@ pg_mb_radix_conv(const pg_mb_radix_tree *rt, * is applied. An error is raised if no match is found. * * See pg_wchar.h for more details about the data structures used here. + * + * Returns the number of input bytes consumed. If noError is true, this can + * be less than 'len'. */ -void +int UtfToLocal(const unsigned char *utf, int len, unsigned char *iso, const pg_mb_radix_tree *map, const pg_utf_to_local_combined *cmap, int cmapsize, utf_local_conversion_func conv_func, - int encoding) + int encoding, bool noError) { uint32 iutf; int l; const pg_utf_to_local_combined *cp; + const unsigned char *start = utf; if (!PG_VALID_ENCODING(encoding)) ereport(ERROR, @@ -505,10 +584,19 @@ UtfToLocal(const unsigned char *utf, int len, l = pg_utf_mblen(utf); if (len < l) + { + /* need more data to decide if this is a combined char */ + utf -= l_save; break; + } if (!pg_utf8_islegal(utf, l)) + { + if (!noError) + report_invalid_encoding(PG_UTF8, (const char *) utf, len); + utf -= l_save; break; + } /* We assume ASCII character cannot be in combined map */ if (l > 1) @@ -584,15 +672,20 @@ UtfToLocal(const unsigned char *utf, int len, } /* failed to translate this character */ + utf -= l; + if (noError) + break; report_untranslatable_char(PG_UTF8, encoding, - (const char *) (utf - l), len); + (const char *) utf, len); } /* if we broke out of loop early, must be invalid input */ - if (len > 0) + if (len > 0 && !noError) report_invalid_encoding(PG_UTF8, (const char *) utf, len); *iso = '\0'; + + return utf - start; } /* @@ -616,18 +709,23 @@ UtfToLocal(const unsigned char *utf, int len, * (if provided) is applied. An error is raised if no match is found. * * See pg_wchar.h for more details about the data structures used here. + * + * Returns the number of input bytes consumed. If noError is true, this can + * be less than 'len'. */ -void +int LocalToUtf(const unsigned char *iso, int len, unsigned char *utf, const pg_mb_radix_tree *map, const pg_local_to_utf_combined *cmap, int cmapsize, utf_local_conversion_func conv_func, - int encoding) + int encoding, + bool noError) { uint32 iiso; int l; const pg_local_to_utf_combined *cp; + const unsigned char *start = iso; if (!PG_VALID_ENCODING(encoding)) ereport(ERROR, @@ -723,13 +821,18 @@ LocalToUtf(const unsigned char *iso, int len, } /* failed to translate this character */ + iso -= l; + if (noError) + break; report_untranslatable_char(encoding, PG_UTF8, - (const char *) (iso - l), len); + (const char *) iso, len); } /* if we broke out of loop early, must be invalid input */ - if (len > 0) + if (len > 0 && !noError) report_invalid_encoding(encoding, (const char *) iso, len); *utf = '\0'; + + return iso - start; } diff --git a/src/backend/utils/mb/conversion_procs/cyrillic_and_mic/cyrillic_and_mic.c b/src/backend/utils/mb/conversion_procs/cyrillic_and_mic/cyrillic_and_mic.c index 4c5b02654de..368c2deb5e4 100644 --- a/src/backend/utils/mb/conversion_procs/cyrillic_and_mic/cyrillic_and_mic.c +++ b/src/backend/utils/mb/conversion_procs/cyrillic_and_mic/cyrillic_and_mic.c @@ -44,8 +44,11 @@ PG_FUNCTION_INFO_V1(win866_to_iso); * INTEGER, -- destination encoding id * CSTRING, -- source string (null terminated C string) * CSTRING, -- destination string (null terminated C string) - * INTEGER -- source string length - * ) returns VOID; + * INTEGER, -- source string length + * BOOL -- if true, don't throw an error if conversion fails + * ) returns INTEGER; + * + * Returns the number of bytes successfully converted. * ---------- */ @@ -306,12 +309,14 @@ koi8r_to_mic(PG_FUNCTION_ARGS) unsigned char *src = (unsigned char *) PG_GETARG_CSTRING(2); unsigned char *dest = (unsigned char *) PG_GETARG_CSTRING(3); int len = PG_GETARG_INT32(4); + bool noError = PG_GETARG_BOOL(5); + int converted; CHECK_ENCODING_CONVERSION_ARGS(PG_KOI8R, PG_MULE_INTERNAL); - latin2mic(src, dest, len, LC_KOI8_R, PG_KOI8R); + converted = latin2mic(src, dest, len, LC_KOI8_R, PG_KOI8R, noError); - PG_RETURN_VOID(); + PG_RETURN_INT32(converted); } Datum @@ -320,12 +325,14 @@ mic_to_koi8r(PG_FUNCTION_ARGS) unsigned char *src = (unsigned char *) PG_GETARG_CSTRING(2); unsigned char *dest = (unsigned char *) PG_GETARG_CSTRING(3); int len = PG_GETARG_INT32(4); + bool noError = PG_GETARG_BOOL(5); + int converted; CHECK_ENCODING_CONVERSION_ARGS(PG_MULE_INTERNAL, PG_KOI8R); - mic2latin(src, dest, len, LC_KOI8_R, PG_KOI8R); + converted = mic2latin(src, dest, len, LC_KOI8_R, PG_KOI8R, noError); - PG_RETURN_VOID(); + PG_RETURN_INT32(converted); } Datum @@ -334,12 +341,14 @@ iso_to_mic(PG_FUNCTION_ARGS) unsigned char *src = (unsigned char *) PG_GETARG_CSTRING(2); unsigned char *dest = (unsigned char *) PG_GETARG_CSTRING(3); int len = PG_GETARG_INT32(4); + bool noError = PG_GETARG_BOOL(5); + int converted; CHECK_ENCODING_CONVERSION_ARGS(PG_ISO_8859_5, PG_MULE_INTERNAL); - latin2mic_with_table(src, dest, len, LC_KOI8_R, PG_ISO_8859_5, iso2koi); + converted = latin2mic_with_table(src, dest, len, LC_KOI8_R, PG_ISO_8859_5, iso2koi, noError); - PG_RETURN_VOID(); + PG_RETURN_INT32(converted); } Datum @@ -348,12 +357,14 @@ mic_to_iso(PG_FUNCTION_ARGS) unsigned char *src = (unsigned char *) PG_GETARG_CSTRING(2); unsigned char *dest = (unsigned char *) PG_GETARG_CSTRING(3); int len = PG_GETARG_INT32(4); + bool noError = PG_GETARG_BOOL(5); + int converted; CHECK_ENCODING_CONVERSION_ARGS(PG_MULE_INTERNAL, PG_ISO_8859_5); - mic2latin_with_table(src, dest, len, LC_KOI8_R, PG_ISO_8859_5, koi2iso); + converted = mic2latin_with_table(src, dest, len, LC_KOI8_R, PG_ISO_8859_5, koi2iso, noError); - PG_RETURN_VOID(); + PG_RETURN_INT32(converted); } Datum @@ -362,12 +373,14 @@ win1251_to_mic(PG_FUNCTION_ARGS) unsigned char *src = (unsigned char *) PG_GETARG_CSTRING(2); unsigned char *dest = (unsigned char *) PG_GETARG_CSTRING(3); int len = PG_GETARG_INT32(4); + bool noError = PG_GETARG_BOOL(5); + int converted; CHECK_ENCODING_CONVERSION_ARGS(PG_WIN1251, PG_MULE_INTERNAL); - latin2mic_with_table(src, dest, len, LC_KOI8_R, PG_WIN1251, win12512koi); + converted = latin2mic_with_table(src, dest, len, LC_KOI8_R, PG_WIN1251, win12512koi, noError); - PG_RETURN_VOID(); + PG_RETURN_INT32(converted); } Datum @@ -376,12 +389,14 @@ mic_to_win1251(PG_FUNCTION_ARGS) unsigned char *src = (unsigned char *) PG_GETARG_CSTRING(2); unsigned char *dest = (unsigned char *) PG_GETARG_CSTRING(3); int len = PG_GETARG_INT32(4); + bool noError = PG_GETARG_BOOL(5); + int converted; CHECK_ENCODING_CONVERSION_ARGS(PG_MULE_INTERNAL, PG_WIN1251); - mic2latin_with_table(src, dest, len, LC_KOI8_R, PG_WIN1251, koi2win1251); + converted = mic2latin_with_table(src, dest, len, LC_KOI8_R, PG_WIN1251, koi2win1251, noError); - PG_RETURN_VOID(); + PG_RETURN_INT32(converted); } Datum @@ -390,12 +405,14 @@ win866_to_mic(PG_FUNCTION_ARGS) unsigned char *src = (unsigned char *) PG_GETARG_CSTRING(2); unsigned char *dest = (unsigned char *) PG_GETARG_CSTRING(3); int len = PG_GETARG_INT32(4); + bool noError = PG_GETARG_BOOL(5); + int converted; CHECK_ENCODING_CONVERSION_ARGS(PG_WIN866, PG_MULE_INTERNAL); - latin2mic_with_table(src, dest, len, LC_KOI8_R, PG_WIN866, win8662koi); + converted = latin2mic_with_table(src, dest, len, LC_KOI8_R, PG_WIN866, win8662koi, noError); - PG_RETURN_VOID(); + PG_RETURN_INT32(converted); } Datum @@ -404,12 +421,14 @@ mic_to_win866(PG_FUNCTION_ARGS) unsigned char *src = (unsigned char *) PG_GETARG_CSTRING(2); unsigned char *dest = (unsigned char *) PG_GETARG_CSTRING(3); int len = PG_GETARG_INT32(4); + bool noError = PG_GETARG_BOOL(5); + int converted; CHECK_ENCODING_CONVERSION_ARGS(PG_MULE_INTERNAL, PG_WIN866); - mic2latin_with_table(src, dest, len, LC_KOI8_R, PG_WIN866, koi2win866); + converted = mic2latin_with_table(src, dest, len, LC_KOI8_R, PG_WIN866, koi2win866, noError); - PG_RETURN_VOID(); + PG_RETURN_INT32(converted); } Datum @@ -418,12 +437,14 @@ koi8r_to_win1251(PG_FUNCTION_ARGS) unsigned char *src = (unsigned char *) PG_GETARG_CSTRING(2); unsigned char *dest = (unsigned char *) PG_GETARG_CSTRING(3); int len = PG_GETARG_INT32(4); + bool noError = PG_GETARG_BOOL(5); + int converted; CHECK_ENCODING_CONVERSION_ARGS(PG_KOI8R, PG_WIN1251); - local2local(src, dest, len, PG_KOI8R, PG_WIN1251, koi2win1251); + converted = local2local(src, dest, len, PG_KOI8R, PG_WIN1251, koi2win1251, noError); - PG_RETURN_VOID(); + PG_RETURN_INT32(converted); } Datum @@ -432,12 +453,14 @@ win1251_to_koi8r(PG_FUNCTION_ARGS) unsigned char *src = (unsigned char *) PG_GETARG_CSTRING(2); unsigned char *dest = (unsigned char *) PG_GETARG_CSTRING(3); int len = PG_GETARG_INT32(4); + bool noError = PG_GETARG_BOOL(5); + int converted; CHECK_ENCODING_CONVERSION_ARGS(PG_WIN1251, PG_KOI8R); - local2local(src, dest, len, PG_WIN1251, PG_KOI8R, win12512koi); + converted = local2local(src, dest, len, PG_WIN1251, PG_KOI8R, win12512koi, noError); - PG_RETURN_VOID(); + PG_RETURN_INT32(converted); } Datum @@ -446,12 +469,14 @@ koi8r_to_win866(PG_FUNCTION_ARGS) unsigned char *src = (unsigned char *) PG_GETARG_CSTRING(2); unsigned char *dest = (unsigned char *) PG_GETARG_CSTRING(3); int len = PG_GETARG_INT32(4); + bool noError = PG_GETARG_BOOL(5); + int converted; CHECK_ENCODING_CONVERSION_ARGS(PG_KOI8R, PG_WIN866); - local2local(src, dest, len, PG_KOI8R, PG_WIN866, koi2win866); + converted = local2local(src, dest, len, PG_KOI8R, PG_WIN866, koi2win866, noError); - PG_RETURN_VOID(); + PG_RETURN_INT32(converted); } Datum @@ -460,12 +485,14 @@ win866_to_koi8r(PG_FUNCTION_ARGS) unsigned char *src = (unsigned char *) PG_GETARG_CSTRING(2); unsigned char *dest = (unsigned char *) PG_GETARG_CSTRING(3); int len = PG_GETARG_INT32(4); + bool noError = PG_GETARG_BOOL(5); + int converted; CHECK_ENCODING_CONVERSION_ARGS(PG_WIN866, PG_KOI8R); - local2local(src, dest, len, PG_WIN866, PG_KOI8R, win8662koi); + converted = local2local(src, dest, len, PG_WIN866, PG_KOI8R, win8662koi, noError); - PG_RETURN_VOID(); + PG_RETURN_INT32(converted); } Datum @@ -474,12 +501,14 @@ win866_to_win1251(PG_FUNCTION_ARGS) unsigned char *src = (unsigned char *) PG_GETARG_CSTRING(2); unsigned char *dest = (unsigned char *) PG_GETARG_CSTRING(3); int len = PG_GETARG_INT32(4); + bool noError = PG_GETARG_BOOL(5); + int converted; CHECK_ENCODING_CONVERSION_ARGS(PG_WIN866, PG_WIN1251); - local2local(src, dest, len, PG_WIN866, PG_WIN1251, win8662win1251); + converted = local2local(src, dest, len, PG_WIN866, PG_WIN1251, win8662win1251, noError); - PG_RETURN_VOID(); + PG_RETURN_INT32(converted); } Datum @@ -488,12 +517,14 @@ win1251_to_win866(PG_FUNCTION_ARGS) unsigned char *src = (unsigned char *) PG_GETARG_CSTRING(2); unsigned char *dest = (unsigned char *) PG_GETARG_CSTRING(3); int len = PG_GETARG_INT32(4); + bool noError = PG_GETARG_BOOL(5); + int converted; CHECK_ENCODING_CONVERSION_ARGS(PG_WIN1251, PG_WIN866); - local2local(src, dest, len, PG_WIN1251, PG_WIN866, win12512win866); + converted = local2local(src, dest, len, PG_WIN1251, PG_WIN866, win12512win866, noError); - PG_RETURN_VOID(); + PG_RETURN_INT32(converted); } Datum @@ -502,12 +533,14 @@ iso_to_koi8r(PG_FUNCTION_ARGS) unsigned char *src = (unsigned char *) PG_GETARG_CSTRING(2); unsigned char *dest = (unsigned char *) PG_GETARG_CSTRING(3); int len = PG_GETARG_INT32(4); + bool noError = PG_GETARG_BOOL(5); + int converted; CHECK_ENCODING_CONVERSION_ARGS(PG_ISO_8859_5, PG_KOI8R); - local2local(src, dest, len, PG_ISO_8859_5, PG_KOI8R, iso2koi); + converted = local2local(src, dest, len, PG_ISO_8859_5, PG_KOI8R, iso2koi, noError); - PG_RETURN_VOID(); + PG_RETURN_INT32(converted); } Datum @@ -516,12 +549,14 @@ koi8r_to_iso(PG_FUNCTION_ARGS) unsigned char *src = (unsigned char *) PG_GETARG_CSTRING(2); unsigned char *dest = (unsigned char *) PG_GETARG_CSTRING(3); int len = PG_GETARG_INT32(4); + bool noError = PG_GETARG_BOOL(5); + int converted; CHECK_ENCODING_CONVERSION_ARGS(PG_KOI8R, PG_ISO_8859_5); - local2local(src, dest, len, PG_KOI8R, PG_ISO_8859_5, koi2iso); + converted = local2local(src, dest, len, PG_KOI8R, PG_ISO_8859_5, koi2iso, noError); - PG_RETURN_VOID(); + PG_RETURN_INT32(converted); } Datum @@ -530,12 +565,14 @@ iso_to_win1251(PG_FUNCTION_ARGS) unsigned char *src = (unsigned char *) PG_GETARG_CSTRING(2); unsigned char *dest = (unsigned char *) PG_GETARG_CSTRING(3); int len = PG_GETARG_INT32(4); + bool noError = PG_GETARG_BOOL(5); + int converted; CHECK_ENCODING_CONVERSION_ARGS(PG_ISO_8859_5, PG_WIN1251); - local2local(src, dest, len, PG_ISO_8859_5, PG_WIN1251, iso2win1251); + converted = local2local(src, dest, len, PG_ISO_8859_5, PG_WIN1251, iso2win1251, noError); - PG_RETURN_VOID(); + PG_RETURN_INT32(converted); } Datum @@ -544,12 +581,14 @@ win1251_to_iso(PG_FUNCTION_ARGS) unsigned char *src = (unsigned char *) PG_GETARG_CSTRING(2); unsigned char *dest = (unsigned char *) PG_GETARG_CSTRING(3); int len = PG_GETARG_INT32(4); + bool noError = PG_GETARG_BOOL(5); + int converted; CHECK_ENCODING_CONVERSION_ARGS(PG_WIN1251, PG_ISO_8859_5); - local2local(src, dest, len, PG_WIN1251, PG_ISO_8859_5, win12512iso); + converted = local2local(src, dest, len, PG_WIN1251, PG_ISO_8859_5, win12512iso, noError); - PG_RETURN_VOID(); + PG_RETURN_INT32(converted); } Datum @@ -558,12 +597,14 @@ iso_to_win866(PG_FUNCTION_ARGS) unsigned char *src = (unsigned char *) PG_GETARG_CSTRING(2); unsigned char *dest = (unsigned char *) PG_GETARG_CSTRING(3); int len = PG_GETARG_INT32(4); + bool noError = PG_GETARG_BOOL(5); + int converted; CHECK_ENCODING_CONVERSION_ARGS(PG_ISO_8859_5, PG_WIN866); - local2local(src, dest, len, PG_ISO_8859_5, PG_WIN866, iso2win866); + converted = local2local(src, dest, len, PG_ISO_8859_5, PG_WIN866, iso2win866, noError); - PG_RETURN_VOID(); + PG_RETURN_INT32(converted); } Datum @@ -572,10 +613,12 @@ win866_to_iso(PG_FUNCTION_ARGS) unsigned char *src = (unsigned char *) PG_GETARG_CSTRING(2); unsigned char *dest = (unsigned char *) PG_GETARG_CSTRING(3); int len = PG_GETARG_INT32(4); + bool noError = PG_GETARG_BOOL(5); + int converted; CHECK_ENCODING_CONVERSION_ARGS(PG_WIN866, PG_ISO_8859_5); - local2local(src, dest, len, PG_WIN866, PG_ISO_8859_5, win8662iso); + converted = local2local(src, dest, len, PG_WIN866, PG_ISO_8859_5, win8662iso, noError); - PG_RETURN_VOID(); + PG_RETURN_INT32(converted); } diff --git a/src/backend/utils/mb/conversion_procs/euc2004_sjis2004/euc2004_sjis2004.c b/src/backend/utils/mb/conversion_procs/euc2004_sjis2004/euc2004_sjis2004.c index 4d7fb116cfd..a3fd35bd406 100644 --- a/src/backend/utils/mb/conversion_procs/euc2004_sjis2004/euc2004_sjis2004.c +++ b/src/backend/utils/mb/conversion_procs/euc2004_sjis2004/euc2004_sjis2004.c @@ -19,8 +19,8 @@ PG_MODULE_MAGIC; PG_FUNCTION_INFO_V1(euc_jis_2004_to_shift_jis_2004); PG_FUNCTION_INFO_V1(shift_jis_2004_to_euc_jis_2004); -static void euc_jis_20042shift_jis_2004(const unsigned char *euc, unsigned char *p, int len); -static void shift_jis_20042euc_jis_2004(const unsigned char *sjis, unsigned char *p, int len); +static int euc_jis_20042shift_jis_2004(const unsigned char *euc, unsigned char *p, int len, bool noError); +static int shift_jis_20042euc_jis_2004(const unsigned char *sjis, unsigned char *p, int len, bool noError); /* ---------- * conv_proc( @@ -28,8 +28,11 @@ static void shift_jis_20042euc_jis_2004(const unsigned char *sjis, unsigned char * INTEGER, -- destination encoding id * CSTRING, -- source string (null terminated C string) * CSTRING, -- destination string (null terminated C string) - * INTEGER -- source string length - * ) returns VOID; + * INTEGER, -- source string length + * BOOL -- if true, don't throw an error if conversion fails + * ) returns INTEGER; + * + * Returns the number of bytes successfully converted. * ---------- */ @@ -39,12 +42,14 @@ euc_jis_2004_to_shift_jis_2004(PG_FUNCTION_ARGS) unsigned char *src = (unsigned char *) PG_GETARG_CSTRING(2); unsigned char *dest = (unsigned char *) PG_GETARG_CSTRING(3); int len = PG_GETARG_INT32(4); + bool noError = PG_GETARG_BOOL(5); + int converted; CHECK_ENCODING_CONVERSION_ARGS(PG_EUC_JIS_2004, PG_SHIFT_JIS_2004); - euc_jis_20042shift_jis_2004(src, dest, len); + converted = euc_jis_20042shift_jis_2004(src, dest, len, noError); - PG_RETURN_VOID(); + PG_RETURN_INT32(converted); } Datum @@ -53,20 +58,23 @@ shift_jis_2004_to_euc_jis_2004(PG_FUNCTION_ARGS) unsigned char *src = (unsigned char *) PG_GETARG_CSTRING(2); unsigned char *dest = (unsigned char *) PG_GETARG_CSTRING(3); int len = PG_GETARG_INT32(4); + bool noError = PG_GETARG_BOOL(5); + int converted; CHECK_ENCODING_CONVERSION_ARGS(PG_SHIFT_JIS_2004, PG_EUC_JIS_2004); - shift_jis_20042euc_jis_2004(src, dest, len); + converted = shift_jis_20042euc_jis_2004(src, dest, len, noError); - PG_RETURN_VOID(); + PG_RETURN_INT32(converted); } /* * EUC_JIS_2004 -> SHIFT_JIS_2004 */ -static void -euc_jis_20042shift_jis_2004(const unsigned char *euc, unsigned char *p, int len) +static int +euc_jis_20042shift_jis_2004(const unsigned char *euc, unsigned char *p, int len, bool noError) { + const unsigned char *start = euc; int c1, ku, ten; @@ -79,8 +87,12 @@ euc_jis_20042shift_jis_2004(const unsigned char *euc, unsigned char *p, int len) { /* ASCII */ if (c1 == 0) + { + if (noError) + break; report_invalid_encoding(PG_EUC_JIS_2004, (const char *) euc, len); + } *p++ = c1; euc++; len--; @@ -90,8 +102,12 @@ euc_jis_20042shift_jis_2004(const unsigned char *euc, unsigned char *p, int len) l = pg_encoding_verifymbchar(PG_EUC_JIS_2004, (const char *) euc, len); if (l < 0) + { + if (noError) + break; report_invalid_encoding(PG_EUC_JIS_2004, (const char *) euc, len); + } if (c1 == SS2 && l == 2) /* JIS X 0201 kana? */ { @@ -121,8 +137,12 @@ euc_jis_20042shift_jis_2004(const unsigned char *euc, unsigned char *p, int len) *p++ = (ku + 0x19b) >> 1; } else + { + if (noError) + break; report_invalid_encoding(PG_EUC_JIS_2004, (const char *) euc, len); + } } if (ku % 2) @@ -132,8 +152,12 @@ euc_jis_20042shift_jis_2004(const unsigned char *euc, unsigned char *p, int len) else if (ten >= 64 && ten <= 94) *p++ = ten + 0x40; else + { + if (noError) + break; report_invalid_encoding(PG_EUC_JIS_2004, (const char *) euc, len); + } } else *p++ = ten + 0x9e; @@ -149,8 +173,12 @@ euc_jis_20042shift_jis_2004(const unsigned char *euc, unsigned char *p, int len) else if (ku >= 63 && ku <= 94) *p++ = (ku + 0x181) >> 1; else + { + if (noError) + break; report_invalid_encoding(PG_EUC_JIS_2004, (const char *) euc, len); + } if (ku % 2) { @@ -159,20 +187,30 @@ euc_jis_20042shift_jis_2004(const unsigned char *euc, unsigned char *p, int len) else if (ten >= 64 && ten <= 94) *p++ = ten + 0x40; else + { + if (noError) + break; report_invalid_encoding(PG_EUC_JIS_2004, (const char *) euc, len); + } } else *p++ = ten + 0x9e; } else + { + if (noError) + break; report_invalid_encoding(PG_EUC_JIS_2004, (const char *) euc, len); + } euc += l; len -= l; } *p = '\0'; + + return euc - start; } /* @@ -212,9 +250,10 @@ get_ten(int b, int *ku) * SHIFT_JIS_2004 ---> EUC_JIS_2004 */ -static void -shift_jis_20042euc_jis_2004(const unsigned char *sjis, unsigned char *p, int len) +static int +shift_jis_20042euc_jis_2004(const unsigned char *sjis, unsigned char *p, int len, bool noError) { + const unsigned char *start = sjis; int c1; int ku, ten, @@ -230,8 +269,12 @@ shift_jis_20042euc_jis_2004(const unsigned char *sjis, unsigned char *p, int len { /* ASCII */ if (c1 == 0) + { + if (noError) + break; report_invalid_encoding(PG_SHIFT_JIS_2004, (const char *) sjis, len); + } *p++ = c1; sjis++; len--; @@ -241,8 +284,12 @@ shift_jis_20042euc_jis_2004(const unsigned char *sjis, unsigned char *p, int len l = pg_encoding_verifymbchar(PG_SHIFT_JIS_2004, (const char *) sjis, len); if (l < 0 || l > len) + { + if (noError) + break; report_invalid_encoding(PG_SHIFT_JIS_2004, (const char *) sjis, len); + } if (c1 >= 0xa1 && c1 <= 0xdf && l == 1) { @@ -266,8 +313,12 @@ shift_jis_20042euc_jis_2004(const unsigned char *sjis, unsigned char *p, int len ku = (c1 << 1) - 0x100; ten = get_ten(c2, &kubun); if (ten < 0) + { + if (noError) + break; report_invalid_encoding(PG_SHIFT_JIS_2004, (const char *) sjis, len); + } ku -= kubun; } else if (c1 >= 0xe0 && c1 <= 0xef) /* plane 1 62ku-94ku */ @@ -275,9 +326,12 @@ shift_jis_20042euc_jis_2004(const unsigned char *sjis, unsigned char *p, int len ku = (c1 << 1) - 0x180; ten = get_ten(c2, &kubun); if (ten < 0) + { + if (noError) + break; report_invalid_encoding(PG_SHIFT_JIS_2004, - (const char *) sjis, len); + } ku -= kubun; } else if (c1 >= 0xf0 && c1 <= 0xf3) /* plane 2 @@ -286,8 +340,12 @@ shift_jis_20042euc_jis_2004(const unsigned char *sjis, unsigned char *p, int len plane = 2; ten = get_ten(c2, &kubun); if (ten < 0) + { + if (noError) + break; report_invalid_encoding(PG_SHIFT_JIS_2004, (const char *) sjis, len); + } switch (c1) { case 0xf0: @@ -309,16 +367,24 @@ shift_jis_20042euc_jis_2004(const unsigned char *sjis, unsigned char *p, int len plane = 2; ten = get_ten(c2, &kubun); if (ten < 0) + { + if (noError) + break; report_invalid_encoding(PG_SHIFT_JIS_2004, (const char *) sjis, len); + } if (c1 == 0xf4 && kubun == 1) ku = 15; else ku = (c1 << 1) - 0x19a - kubun; } else + { + if (noError) + break; report_invalid_encoding(PG_SHIFT_JIS_2004, (const char *) sjis, len); + } if (plane == 2) *p++ = SS3; @@ -330,4 +396,6 @@ shift_jis_20042euc_jis_2004(const unsigned char *sjis, unsigned char *p, int len len -= l; } *p = '\0'; + + return sjis - start; } diff --git a/src/backend/utils/mb/conversion_procs/euc_cn_and_mic/euc_cn_and_mic.c b/src/backend/utils/mb/conversion_procs/euc_cn_and_mic/euc_cn_and_mic.c index e9bb896935f..09b3c2e75bf 100644 --- a/src/backend/utils/mb/conversion_procs/euc_cn_and_mic/euc_cn_and_mic.c +++ b/src/backend/utils/mb/conversion_procs/euc_cn_and_mic/euc_cn_and_mic.c @@ -26,13 +26,16 @@ PG_FUNCTION_INFO_V1(mic_to_euc_cn); * INTEGER, -- destination encoding id * CSTRING, -- source string (null terminated C string) * CSTRING, -- destination string (null terminated C string) - * INTEGER -- source string length - * ) returns VOID; + * INTEGER, -- source string length + * BOOL -- if true, don't throw an error if conversion fails + * ) returns INTEGER; + * + * Returns the number of bytes successfully converted. * ---------- */ -static void euc_cn2mic(const unsigned char *euc, unsigned char *p, int len); -static void mic2euc_cn(const unsigned char *mic, unsigned char *p, int len); +static int euc_cn2mic(const unsigned char *euc, unsigned char *p, int len, bool noError); +static int mic2euc_cn(const unsigned char *mic, unsigned char *p, int len, bool noError); Datum euc_cn_to_mic(PG_FUNCTION_ARGS) @@ -40,12 +43,14 @@ euc_cn_to_mic(PG_FUNCTION_ARGS) unsigned char *src = (unsigned char *) PG_GETARG_CSTRING(2); unsigned char *dest = (unsigned char *) PG_GETARG_CSTRING(3); int len = PG_GETARG_INT32(4); + bool noError = PG_GETARG_BOOL(5); + int converted; CHECK_ENCODING_CONVERSION_ARGS(PG_EUC_CN, PG_MULE_INTERNAL); - euc_cn2mic(src, dest, len); + converted = euc_cn2mic(src, dest, len, noError); - PG_RETURN_VOID(); + PG_RETURN_INT32(converted); } Datum @@ -54,20 +59,23 @@ mic_to_euc_cn(PG_FUNCTION_ARGS) unsigned char *src = (unsigned char *) PG_GETARG_CSTRING(2); unsigned char *dest = (unsigned char *) PG_GETARG_CSTRING(3); int len = PG_GETARG_INT32(4); + bool noError = PG_GETARG_BOOL(5); + int converted; CHECK_ENCODING_CONVERSION_ARGS(PG_MULE_INTERNAL, PG_EUC_CN); - mic2euc_cn(src, dest, len); + converted = mic2euc_cn(src, dest, len, noError); - PG_RETURN_VOID(); + PG_RETURN_INT32(converted); } /* * EUC_CN ---> MIC */ -static void -euc_cn2mic(const unsigned char *euc, unsigned char *p, int len) +static int +euc_cn2mic(const unsigned char *euc, unsigned char *p, int len, bool noError) { + const unsigned char *start = euc; int c1; while (len > 0) @@ -76,7 +84,11 @@ euc_cn2mic(const unsigned char *euc, unsigned char *p, int len) if (IS_HIGHBIT_SET(c1)) { if (len < 2 || !IS_HIGHBIT_SET(euc[1])) + { + if (noError) + break; report_invalid_encoding(PG_EUC_CN, (const char *) euc, len); + } *p++ = LC_GB2312_80; *p++ = c1; *p++ = euc[1]; @@ -86,21 +98,28 @@ euc_cn2mic(const unsigned char *euc, unsigned char *p, int len) else { /* should be ASCII */ if (c1 == 0) + { + if (noError) + break; report_invalid_encoding(PG_EUC_CN, (const char *) euc, len); + } *p++ = c1; euc++; len--; } } *p = '\0'; + + return euc - start; } /* * MIC ---> EUC_CN */ -static void -mic2euc_cn(const unsigned char *mic, unsigned char *p, int len) +static int +mic2euc_cn(const unsigned char *mic, unsigned char *p, int len, bool noError) { + const unsigned char *start = mic; int c1; while (len > 0) @@ -109,11 +128,19 @@ mic2euc_cn(const unsigned char *mic, unsigned char *p, int len) if (IS_HIGHBIT_SET(c1)) { if (c1 != LC_GB2312_80) + { + if (noError) + break; report_untranslatable_char(PG_MULE_INTERNAL, PG_EUC_CN, (const char *) mic, len); + } if (len < 3 || !IS_HIGHBIT_SET(mic[1]) || !IS_HIGHBIT_SET(mic[2])) + { + if (noError) + break; report_invalid_encoding(PG_MULE_INTERNAL, (const char *) mic, len); + } mic++; *p++ = *mic++; *p++ = *mic++; @@ -122,12 +149,18 @@ mic2euc_cn(const unsigned char *mic, unsigned char *p, int len) else { /* should be ASCII */ if (c1 == 0) + { + if (noError) + break; report_invalid_encoding(PG_MULE_INTERNAL, (const char *) mic, len); + } *p++ = c1; mic++; len--; } } *p = '\0'; + + return mic - start; } diff --git a/src/backend/utils/mb/conversion_procs/euc_jp_and_sjis/euc_jp_and_sjis.c b/src/backend/utils/mb/conversion_procs/euc_jp_and_sjis/euc_jp_and_sjis.c index 5059f917a98..2e68708893d 100644 --- a/src/backend/utils/mb/conversion_procs/euc_jp_and_sjis/euc_jp_and_sjis.c +++ b/src/backend/utils/mb/conversion_procs/euc_jp_and_sjis/euc_jp_and_sjis.c @@ -42,17 +42,20 @@ PG_FUNCTION_INFO_V1(mic_to_sjis); * INTEGER, -- destination encoding id * CSTRING, -- source string (null terminated C string) * CSTRING, -- destination string (null terminated C string) - * INTEGER -- source string length - * ) returns VOID; + * INTEGER, -- source string length + * BOOL -- if true, don't throw an error if conversion fails + * ) returns INTEGER; + * + * Returns the number of bytes successfully converted. * ---------- */ -static void sjis2mic(const unsigned char *sjis, unsigned char *p, int len); -static void mic2sjis(const unsigned char *mic, unsigned char *p, int len); -static void euc_jp2mic(const unsigned char *euc, unsigned char *p, int len); -static void mic2euc_jp(const unsigned char *mic, unsigned char *p, int len); -static void euc_jp2sjis(const unsigned char *mic, unsigned char *p, int len); -static void sjis2euc_jp(const unsigned char *mic, unsigned char *p, int len); +static int sjis2mic(const unsigned char *sjis, unsigned char *p, int len, bool noError); +static int mic2sjis(const unsigned char *mic, unsigned char *p, int len, bool noError); +static int euc_jp2mic(const unsigned char *euc, unsigned char *p, int len, bool noError); +static int mic2euc_jp(const unsigned char *mic, unsigned char *p, int len, bool noError); +static int euc_jp2sjis(const unsigned char *mic, unsigned char *p, int len, bool noError); +static int sjis2euc_jp(const unsigned char *mic, unsigned char *p, int len, bool noError); Datum euc_jp_to_sjis(PG_FUNCTION_ARGS) @@ -60,12 +63,14 @@ euc_jp_to_sjis(PG_FUNCTION_ARGS) unsigned char *src = (unsigned char *) PG_GETARG_CSTRING(2); unsigned char *dest = (unsigned char *) PG_GETARG_CSTRING(3); int len = PG_GETARG_INT32(4); + bool noError = PG_GETARG_BOOL(5); + int converted; CHECK_ENCODING_CONVERSION_ARGS(PG_EUC_JP, PG_SJIS); - euc_jp2sjis(src, dest, len); + converted = euc_jp2sjis(src, dest, len, noError); - PG_RETURN_VOID(); + PG_RETURN_INT32(converted); } Datum @@ -74,12 +79,14 @@ sjis_to_euc_jp(PG_FUNCTION_ARGS) unsigned char *src = (unsigned char *) PG_GETARG_CSTRING(2); unsigned char *dest = (unsigned char *) PG_GETARG_CSTRING(3); int len = PG_GETARG_INT32(4); + bool noError = PG_GETARG_BOOL(5); + int converted; CHECK_ENCODING_CONVERSION_ARGS(PG_SJIS, PG_EUC_JP); - sjis2euc_jp(src, dest, len); + converted = sjis2euc_jp(src, dest, len, noError); - PG_RETURN_VOID(); + PG_RETURN_INT32(converted); } Datum @@ -88,12 +95,14 @@ euc_jp_to_mic(PG_FUNCTION_ARGS) unsigned char *src = (unsigned char *) PG_GETARG_CSTRING(2); unsigned char *dest = (unsigned char *) PG_GETARG_CSTRING(3); int len = PG_GETARG_INT32(4); + bool noError = PG_GETARG_BOOL(5); + int converted; CHECK_ENCODING_CONVERSION_ARGS(PG_EUC_JP, PG_MULE_INTERNAL); - euc_jp2mic(src, dest, len); + converted = euc_jp2mic(src, dest, len, noError); - PG_RETURN_VOID(); + PG_RETURN_INT32(converted); } Datum @@ -102,12 +111,14 @@ mic_to_euc_jp(PG_FUNCTION_ARGS) unsigned char *src = (unsigned char *) PG_GETARG_CSTRING(2); unsigned char *dest = (unsigned char *) PG_GETARG_CSTRING(3); int len = PG_GETARG_INT32(4); + bool noError = PG_GETARG_BOOL(5); + int converted; CHECK_ENCODING_CONVERSION_ARGS(PG_MULE_INTERNAL, PG_EUC_JP); - mic2euc_jp(src, dest, len); + converted = mic2euc_jp(src, dest, len, noError); - PG_RETURN_VOID(); + PG_RETURN_INT32(converted); } Datum @@ -116,12 +127,14 @@ sjis_to_mic(PG_FUNCTION_ARGS) unsigned char *src = (unsigned char *) PG_GETARG_CSTRING(2); unsigned char *dest = (unsigned char *) PG_GETARG_CSTRING(3); int len = PG_GETARG_INT32(4); + bool noError = PG_GETARG_BOOL(5); + int converted; CHECK_ENCODING_CONVERSION_ARGS(PG_SJIS, PG_MULE_INTERNAL); - sjis2mic(src, dest, len); + converted = sjis2mic(src, dest, len, noError); - PG_RETURN_VOID(); + PG_RETURN_INT32(converted); } Datum @@ -130,20 +143,23 @@ mic_to_sjis(PG_FUNCTION_ARGS) unsigned char *src = (unsigned char *) PG_GETARG_CSTRING(2); unsigned char *dest = (unsigned char *) PG_GETARG_CSTRING(3); int len = PG_GETARG_INT32(4); + bool noError = PG_GETARG_BOOL(5); + int converted; CHECK_ENCODING_CONVERSION_ARGS(PG_MULE_INTERNAL, PG_SJIS); - mic2sjis(src, dest, len); + converted = mic2sjis(src, dest, len, noError); - PG_RETURN_VOID(); + PG_RETURN_INT32(converted); } /* * SJIS ---> MIC */ -static void -sjis2mic(const unsigned char *sjis, unsigned char *p, int len) +static int +sjis2mic(const unsigned char *sjis, unsigned char *p, int len, bool noError) { + const unsigned char *start = sjis; int c1, c2, i, @@ -167,7 +183,11 @@ sjis2mic(const unsigned char *sjis, unsigned char *p, int len) * JIS X0208, X0212, user defined extended characters */ if (len < 2 || !ISSJISHEAD(c1) || !ISSJISTAIL(sjis[1])) + { + if (noError) + break; report_invalid_encoding(PG_SJIS, (const char *) sjis, len); + } c2 = sjis[1]; k = (c1 << 8) + c2; if (k >= 0xed40 && k < 0xf040) @@ -257,21 +277,28 @@ sjis2mic(const unsigned char *sjis, unsigned char *p, int len) else { /* should be ASCII */ if (c1 == 0) + { + if (noError) + break; report_invalid_encoding(PG_SJIS, (const char *) sjis, len); + } *p++ = c1; sjis++; len--; } } *p = '\0'; + + return sjis - start; } /* * MIC ---> SJIS */ -static void -mic2sjis(const unsigned char *mic, unsigned char *p, int len) +static int +mic2sjis(const unsigned char *mic, unsigned char *p, int len, bool noError) { + const unsigned char *start = mic; int c1, c2, k, @@ -284,8 +311,12 @@ mic2sjis(const unsigned char *mic, unsigned char *p, int len) { /* ASCII */ if (c1 == 0) + { + if (noError) + break; report_invalid_encoding(PG_MULE_INTERNAL, (const char *) mic, len); + } *p++ = c1; mic++; len--; @@ -293,8 +324,12 @@ mic2sjis(const unsigned char *mic, unsigned char *p, int len) } l = pg_encoding_verifymbchar(PG_MULE_INTERNAL, (const char *) mic, len); if (l < 0) + { + if (noError) + break; report_invalid_encoding(PG_MULE_INTERNAL, (const char *) mic, len); + } if (c1 == LC_JISX0201K) *p++ = mic[1]; else if (c1 == LC_JISX0208) @@ -350,20 +385,27 @@ mic2sjis(const unsigned char *mic, unsigned char *p, int len) } } else + { + if (noError) + break; report_untranslatable_char(PG_MULE_INTERNAL, PG_SJIS, (const char *) mic, len); + } mic += l; len -= l; } *p = '\0'; + + return mic - start; } /* * EUC_JP ---> MIC */ -static void -euc_jp2mic(const unsigned char *euc, unsigned char *p, int len) +static int +euc_jp2mic(const unsigned char *euc, unsigned char *p, int len, bool noError) { + const unsigned char *start = euc; int c1; int l; @@ -374,8 +416,12 @@ euc_jp2mic(const unsigned char *euc, unsigned char *p, int len) { /* ASCII */ if (c1 == 0) + { + if (noError) + break; report_invalid_encoding(PG_EUC_JP, (const char *) euc, len); + } *p++ = c1; euc++; len--; @@ -383,8 +429,12 @@ euc_jp2mic(const unsigned char *euc, unsigned char *p, int len) } l = pg_encoding_verifymbchar(PG_EUC_JP, (const char *) euc, len); if (l < 0) + { + if (noError) + break; report_invalid_encoding(PG_EUC_JP, (const char *) euc, len); + } if (c1 == SS2) { /* 1 byte kana? */ *p++ = LC_JISX0201K; @@ -406,14 +456,17 @@ euc_jp2mic(const unsigned char *euc, unsigned char *p, int len) len -= l; } *p = '\0'; + + return euc - start; } /* * MIC ---> EUC_JP */ -static void -mic2euc_jp(const unsigned char *mic, unsigned char *p, int len) +static int +mic2euc_jp(const unsigned char *mic, unsigned char *p, int len, bool noError) { + const unsigned char *start = mic; int c1; int l; @@ -424,8 +477,12 @@ mic2euc_jp(const unsigned char *mic, unsigned char *p, int len) { /* ASCII */ if (c1 == 0) + { + if (noError) + break; report_invalid_encoding(PG_MULE_INTERNAL, (const char *) mic, len); + } *p++ = c1; mic++; len--; @@ -433,8 +490,12 @@ mic2euc_jp(const unsigned char *mic, unsigned char *p, int len) } l = pg_encoding_verifymbchar(PG_MULE_INTERNAL, (const char *) mic, len); if (l < 0) + { + if (noError) + break; report_invalid_encoding(PG_MULE_INTERNAL, (const char *) mic, len); + } if (c1 == LC_JISX0201K) { *p++ = SS2; @@ -452,20 +513,27 @@ mic2euc_jp(const unsigned char *mic, unsigned char *p, int len) *p++ = mic[2]; } else + { + if (noError) + break; report_untranslatable_char(PG_MULE_INTERNAL, PG_EUC_JP, (const char *) mic, len); + } mic += l; len -= l; } *p = '\0'; + + return mic - start; } /* * EUC_JP -> SJIS */ -static void -euc_jp2sjis(const unsigned char *euc, unsigned char *p, int len) +static int +euc_jp2sjis(const unsigned char *euc, unsigned char *p, int len, bool noError) { + const unsigned char *start = euc; int c1, c2, k; @@ -478,8 +546,12 @@ euc_jp2sjis(const unsigned char *euc, unsigned char *p, int len) { /* ASCII */ if (c1 == 0) + { + if (noError) + break; report_invalid_encoding(PG_EUC_JP, (const char *) euc, len); + } *p++ = c1; euc++; len--; @@ -487,8 +559,12 @@ euc_jp2sjis(const unsigned char *euc, unsigned char *p, int len) } l = pg_encoding_verifymbchar(PG_EUC_JP, (const char *) euc, len); if (l < 0) + { + if (noError) + break; report_invalid_encoding(PG_EUC_JP, (const char *) euc, len); + } if (c1 == SS2) { /* hankaku kana? */ @@ -551,14 +627,17 @@ euc_jp2sjis(const unsigned char *euc, unsigned char *p, int len) len -= l; } *p = '\0'; + + return euc - start; } /* * SJIS ---> EUC_JP */ -static void -sjis2euc_jp(const unsigned char *sjis, unsigned char *p, int len) +static int +sjis2euc_jp(const unsigned char *sjis, unsigned char *p, int len, bool noError) { + const unsigned char *start = sjis; int c1, c2, i, @@ -573,8 +652,12 @@ sjis2euc_jp(const unsigned char *sjis, unsigned char *p, int len) { /* ASCII */ if (c1 == 0) + { + if (noError) + break; report_invalid_encoding(PG_SJIS, (const char *) sjis, len); + } *p++ = c1; sjis++; len--; @@ -582,8 +665,12 @@ sjis2euc_jp(const unsigned char *sjis, unsigned char *p, int len) } l = pg_encoding_verifymbchar(PG_SJIS, (const char *) sjis, len); if (l < 0) + { + if (noError) + break; report_invalid_encoding(PG_SJIS, (const char *) sjis, len); + } if (c1 >= 0xa1 && c1 <= 0xdf) { /* JIS X0201 (1 byte kana) */ @@ -680,4 +767,6 @@ sjis2euc_jp(const unsigned char *sjis, unsigned char *p, int len) len -= l; } *p = '\0'; + + return sjis - start; } diff --git a/src/backend/utils/mb/conversion_procs/euc_kr_and_mic/euc_kr_and_mic.c b/src/backend/utils/mb/conversion_procs/euc_kr_and_mic/euc_kr_and_mic.c index ac823d6c270..3b85f0c1861 100644 --- a/src/backend/utils/mb/conversion_procs/euc_kr_and_mic/euc_kr_and_mic.c +++ b/src/backend/utils/mb/conversion_procs/euc_kr_and_mic/euc_kr_and_mic.c @@ -26,13 +26,16 @@ PG_FUNCTION_INFO_V1(mic_to_euc_kr); * INTEGER, -- destination encoding id * CSTRING, -- source string (null terminated C string) * CSTRING, -- destination string (null terminated C string) - * INTEGER -- source string length - * ) returns VOID; + * INTEGER, -- source string length + * BOOL -- if true, don't throw an error if conversion fails + * ) returns INTEGER; + * + * Returns the number of bytes successfully converted. * ---------- */ -static void euc_kr2mic(const unsigned char *euc, unsigned char *p, int len); -static void mic2euc_kr(const unsigned char *mic, unsigned char *p, int len); +static int euc_kr2mic(const unsigned char *euc, unsigned char *p, int len, bool noError); +static int mic2euc_kr(const unsigned char *mic, unsigned char *p, int len, bool noError); Datum euc_kr_to_mic(PG_FUNCTION_ARGS) @@ -40,12 +43,14 @@ euc_kr_to_mic(PG_FUNCTION_ARGS) unsigned char *src = (unsigned char *) PG_GETARG_CSTRING(2); unsigned char *dest = (unsigned char *) PG_GETARG_CSTRING(3); int len = PG_GETARG_INT32(4); + bool noError = PG_GETARG_BOOL(5); + int converted; CHECK_ENCODING_CONVERSION_ARGS(PG_EUC_KR, PG_MULE_INTERNAL); - euc_kr2mic(src, dest, len); + converted = euc_kr2mic(src, dest, len, noError); - PG_RETURN_VOID(); + PG_RETURN_INT32(converted); } Datum @@ -54,20 +59,23 @@ mic_to_euc_kr(PG_FUNCTION_ARGS) unsigned char *src = (unsigned char *) PG_GETARG_CSTRING(2); unsigned char *dest = (unsigned char *) PG_GETARG_CSTRING(3); int len = PG_GETARG_INT32(4); + bool noError = PG_GETARG_BOOL(5); + int converted; CHECK_ENCODING_CONVERSION_ARGS(PG_MULE_INTERNAL, PG_EUC_KR); - mic2euc_kr(src, dest, len); + converted = mic2euc_kr(src, dest, len, noError); - PG_RETURN_VOID(); + PG_RETURN_INT32(converted); } /* * EUC_KR ---> MIC */ -static void -euc_kr2mic(const unsigned char *euc, unsigned char *p, int len) +static int +euc_kr2mic(const unsigned char *euc, unsigned char *p, int len, bool noError) { + const unsigned char *start = euc; int c1; int l; @@ -78,8 +86,12 @@ euc_kr2mic(const unsigned char *euc, unsigned char *p, int len) { l = pg_encoding_verifymbchar(PG_EUC_KR, (const char *) euc, len); if (l != 2) + { + if (noError) + break; report_invalid_encoding(PG_EUC_KR, (const char *) euc, len); + } *p++ = LC_KS5601; *p++ = c1; *p++ = euc[1]; @@ -89,22 +101,29 @@ euc_kr2mic(const unsigned char *euc, unsigned char *p, int len) else { /* should be ASCII */ if (c1 == 0) + { + if (noError) + break; report_invalid_encoding(PG_EUC_KR, (const char *) euc, len); + } *p++ = c1; euc++; len--; } } *p = '\0'; + + return euc - start; } /* * MIC ---> EUC_KR */ -static void -mic2euc_kr(const unsigned char *mic, unsigned char *p, int len) +static int +mic2euc_kr(const unsigned char *mic, unsigned char *p, int len, bool noError) { + const unsigned char *start = mic; int c1; int l; @@ -115,8 +134,12 @@ mic2euc_kr(const unsigned char *mic, unsigned char *p, int len) { /* ASCII */ if (c1 == 0) + { + if (noError) + break; report_invalid_encoding(PG_MULE_INTERNAL, (const char *) mic, len); + } *p++ = c1; mic++; len--; @@ -124,18 +147,28 @@ mic2euc_kr(const unsigned char *mic, unsigned char *p, int len) } l = pg_encoding_verifymbchar(PG_MULE_INTERNAL, (const char *) mic, len); if (l < 0) + { + if (noError) + break; report_invalid_encoding(PG_MULE_INTERNAL, (const char *) mic, len); + } if (c1 == LC_KS5601) { *p++ = mic[1]; *p++ = mic[2]; } else + { + if (noError) + break; report_untranslatable_char(PG_MULE_INTERNAL, PG_EUC_KR, (const char *) mic, len); + } mic += l; len -= l; } *p = '\0'; + + return mic - start; } diff --git a/src/backend/utils/mb/conversion_procs/euc_tw_and_big5/euc_tw_and_big5.c b/src/backend/utils/mb/conversion_procs/euc_tw_and_big5/euc_tw_and_big5.c index 66c242d7f36..4bf8acda99f 100644 --- a/src/backend/utils/mb/conversion_procs/euc_tw_and_big5/euc_tw_and_big5.c +++ b/src/backend/utils/mb/conversion_procs/euc_tw_and_big5/euc_tw_and_big5.c @@ -32,17 +32,20 @@ PG_FUNCTION_INFO_V1(mic_to_big5); * INTEGER, -- destination encoding id * CSTRING, -- source string (null terminated C string) * CSTRING, -- destination string (null terminated C string) - * INTEGER -- source string length - * ) returns VOID; + * INTEGER, -- source string length + * BOOL -- if true, don't throw an error if conversion fails + * ) returns INTEGER; + * + * Returns the number of bytes successfully converted. * ---------- */ -static void euc_tw2big5(const unsigned char *euc, unsigned char *p, int len); -static void big52euc_tw(const unsigned char *euc, unsigned char *p, int len); -static void big52mic(const unsigned char *big5, unsigned char *p, int len); -static void mic2big5(const unsigned char *mic, unsigned char *p, int len); -static void euc_tw2mic(const unsigned char *euc, unsigned char *p, int len); -static void mic2euc_tw(const unsigned char *mic, unsigned char *p, int len); +static int euc_tw2big5(const unsigned char *euc, unsigned char *p, int len, bool noError); +static int big52euc_tw(const unsigned char *euc, unsigned char *p, int len, bool noError); +static int big52mic(const unsigned char *big5, unsigned char *p, int len, bool noError); +static int mic2big5(const unsigned char *mic, unsigned char *p, int len, bool noError); +static int euc_tw2mic(const unsigned char *euc, unsigned char *p, int len, bool noError); +static int mic2euc_tw(const unsigned char *mic, unsigned char *p, int len, bool noError); Datum euc_tw_to_big5(PG_FUNCTION_ARGS) @@ -50,12 +53,14 @@ euc_tw_to_big5(PG_FUNCTION_ARGS) unsigned char *src = (unsigned char *) PG_GETARG_CSTRING(2); unsigned char *dest = (unsigned char *) PG_GETARG_CSTRING(3); int len = PG_GETARG_INT32(4); + bool noError = PG_GETARG_BOOL(5); + int converted; CHECK_ENCODING_CONVERSION_ARGS(PG_EUC_TW, PG_BIG5); - euc_tw2big5(src, dest, len); + converted = euc_tw2big5(src, dest, len, noError); - PG_RETURN_VOID(); + PG_RETURN_INT32(converted); } Datum @@ -64,12 +69,14 @@ big5_to_euc_tw(PG_FUNCTION_ARGS) unsigned char *src = (unsigned char *) PG_GETARG_CSTRING(2); unsigned char *dest = (unsigned char *) PG_GETARG_CSTRING(3); int len = PG_GETARG_INT32(4); + bool noError = PG_GETARG_BOOL(5); + int converted; CHECK_ENCODING_CONVERSION_ARGS(PG_BIG5, PG_EUC_TW); - big52euc_tw(src, dest, len); + converted = big52euc_tw(src, dest, len, noError); - PG_RETURN_VOID(); + PG_RETURN_INT32(converted); } Datum @@ -78,12 +85,14 @@ euc_tw_to_mic(PG_FUNCTION_ARGS) unsigned char *src = (unsigned char *) PG_GETARG_CSTRING(2); unsigned char *dest = (unsigned char *) PG_GETARG_CSTRING(3); int len = PG_GETARG_INT32(4); + bool noError = PG_GETARG_BOOL(5); + int converted; CHECK_ENCODING_CONVERSION_ARGS(PG_EUC_TW, PG_MULE_INTERNAL); - euc_tw2mic(src, dest, len); + converted = euc_tw2mic(src, dest, len, noError); - PG_RETURN_VOID(); + PG_RETURN_INT32(converted); } Datum @@ -92,12 +101,14 @@ mic_to_euc_tw(PG_FUNCTION_ARGS) unsigned char *src = (unsigned char *) PG_GETARG_CSTRING(2); unsigned char *dest = (unsigned char *) PG_GETARG_CSTRING(3); int len = PG_GETARG_INT32(4); + bool noError = PG_GETARG_BOOL(5); + int converted; CHECK_ENCODING_CONVERSION_ARGS(PG_MULE_INTERNAL, PG_EUC_TW); - mic2euc_tw(src, dest, len); + converted = mic2euc_tw(src, dest, len, noError); - PG_RETURN_VOID(); + PG_RETURN_INT32(converted); } Datum @@ -106,12 +117,14 @@ big5_to_mic(PG_FUNCTION_ARGS) unsigned char *src = (unsigned char *) PG_GETARG_CSTRING(2); unsigned char *dest = (unsigned char *) PG_GETARG_CSTRING(3); int len = PG_GETARG_INT32(4); + bool noError = PG_GETARG_BOOL(5); + int converted; CHECK_ENCODING_CONVERSION_ARGS(PG_BIG5, PG_MULE_INTERNAL); - big52mic(src, dest, len); + converted = big52mic(src, dest, len, noError); - PG_RETURN_VOID(); + PG_RETURN_INT32(converted); } Datum @@ -120,21 +133,24 @@ mic_to_big5(PG_FUNCTION_ARGS) unsigned char *src = (unsigned char *) PG_GETARG_CSTRING(2); unsigned char *dest = (unsigned char *) PG_GETARG_CSTRING(3); int len = PG_GETARG_INT32(4); + bool noError = PG_GETARG_BOOL(5); + int converted; CHECK_ENCODING_CONVERSION_ARGS(PG_MULE_INTERNAL, PG_BIG5); - mic2big5(src, dest, len); + converted = mic2big5(src, dest, len, noError); - PG_RETURN_VOID(); + PG_RETURN_INT32(converted); } /* * EUC_TW ---> Big5 */ -static void -euc_tw2big5(const unsigned char *euc, unsigned char *p, int len) +static int +euc_tw2big5(const unsigned char *euc, unsigned char *p, int len, bool noError) { + const unsigned char *start = euc; unsigned char c1; unsigned short big5buf, cnsBuf; @@ -149,8 +165,12 @@ euc_tw2big5(const unsigned char *euc, unsigned char *p, int len) /* Verify and decode the next EUC_TW input character */ l = pg_encoding_verifymbchar(PG_EUC_TW, (const char *) euc, len); if (l < 0) + { + if (noError) + break; report_invalid_encoding(PG_EUC_TW, (const char *) euc, len); + } if (c1 == SS2) { c1 = euc[1]; /* plane No. */ @@ -171,8 +191,12 @@ euc_tw2big5(const unsigned char *euc, unsigned char *p, int len) /* Write it out in Big5 */ big5buf = CNStoBIG5(cnsBuf, lc); if (big5buf == 0) + { + if (noError) + break; report_untranslatable_char(PG_EUC_TW, PG_BIG5, (const char *) euc, len); + } *p++ = (big5buf >> 8) & 0x00ff; *p++ = big5buf & 0x00ff; @@ -182,22 +206,29 @@ euc_tw2big5(const unsigned char *euc, unsigned char *p, int len) else { /* should be ASCII */ if (c1 == 0) + { + if (noError) + break; report_invalid_encoding(PG_EUC_TW, (const char *) euc, len); + } *p++ = c1; euc++; len--; } } *p = '\0'; + + return euc - start; } /* * Big5 ---> EUC_TW */ -static void -big52euc_tw(const unsigned char *big5, unsigned char *p, int len) +static int +big52euc_tw(const unsigned char *big5, unsigned char *p, int len, bool noError) { + const unsigned char *start = big5; unsigned short c1; unsigned short big5buf, cnsBuf; @@ -212,8 +243,12 @@ big52euc_tw(const unsigned char *big5, unsigned char *p, int len) { l = pg_encoding_verifymbchar(PG_BIG5, (const char *) big5, len); if (l < 0) + { + if (noError) + break; report_invalid_encoding(PG_BIG5, (const char *) big5, len); + } big5buf = (c1 << 8) | big5[1]; cnsBuf = BIG5toCNS(big5buf, &lc); @@ -237,8 +272,12 @@ big52euc_tw(const unsigned char *big5, unsigned char *p, int len) *p++ = cnsBuf & 0x00ff; } else + { + if (noError) + break; report_untranslatable_char(PG_BIG5, PG_EUC_TW, (const char *) big5, len); + } big5 += l; len -= l; @@ -256,14 +295,17 @@ big52euc_tw(const unsigned char *big5, unsigned char *p, int len) } } *p = '\0'; + + return big5 - start; } /* * EUC_TW ---> MIC */ -static void -euc_tw2mic(const unsigned char *euc, unsigned char *p, int len) +static int +euc_tw2mic(const unsigned char *euc, unsigned char *p, int len, bool noError) { + const unsigned char *start = euc; int c1; int l; @@ -274,8 +316,12 @@ euc_tw2mic(const unsigned char *euc, unsigned char *p, int len) { l = pg_encoding_verifymbchar(PG_EUC_TW, (const char *) euc, len); if (l < 0) + { + if (noError) + break; report_invalid_encoding(PG_EUC_TW, (const char *) euc, len); + } if (c1 == SS2) { c1 = euc[1]; /* plane No. */ @@ -304,22 +350,29 @@ euc_tw2mic(const unsigned char *euc, unsigned char *p, int len) else { /* should be ASCII */ if (c1 == 0) + { + if (noError) + break; report_invalid_encoding(PG_EUC_TW, (const char *) euc, len); + } *p++ = c1; euc++; len--; } } *p = '\0'; + + return euc - start; } /* * MIC ---> EUC_TW */ -static void -mic2euc_tw(const unsigned char *mic, unsigned char *p, int len) +static int +mic2euc_tw(const unsigned char *mic, unsigned char *p, int len, bool noError) { + const unsigned char *start = mic; int c1; int l; @@ -330,8 +383,12 @@ mic2euc_tw(const unsigned char *mic, unsigned char *p, int len) { /* ASCII */ if (c1 == 0) + { + if (noError) + break; report_invalid_encoding(PG_MULE_INTERNAL, (const char *) mic, len); + } *p++ = c1; mic++; len--; @@ -339,8 +396,12 @@ mic2euc_tw(const unsigned char *mic, unsigned char *p, int len) } l = pg_encoding_verifymbchar(PG_MULE_INTERNAL, (const char *) mic, len); if (l < 0) + { + if (noError) + break; report_invalid_encoding(PG_MULE_INTERNAL, (const char *) mic, len); + } if (c1 == LC_CNS11643_1) { *p++ = mic[1]; @@ -362,20 +423,27 @@ mic2euc_tw(const unsigned char *mic, unsigned char *p, int len) *p++ = mic[3]; } else + { + if (noError) + break; report_untranslatable_char(PG_MULE_INTERNAL, PG_EUC_TW, (const char *) mic, len); + } mic += l; len -= l; } *p = '\0'; + + return mic - start; } /* * Big5 ---> MIC */ -static void -big52mic(const unsigned char *big5, unsigned char *p, int len) +static int +big52mic(const unsigned char *big5, unsigned char *p, int len, bool noError) { + const unsigned char *start = big5; unsigned short c1; unsigned short big5buf, cnsBuf; @@ -389,8 +457,12 @@ big52mic(const unsigned char *big5, unsigned char *p, int len) { /* ASCII */ if (c1 == 0) + { + if (noError) + break; report_invalid_encoding(PG_BIG5, (const char *) big5, len); + } *p++ = c1; big5++; len--; @@ -398,8 +470,12 @@ big52mic(const unsigned char *big5, unsigned char *p, int len) } l = pg_encoding_verifymbchar(PG_BIG5, (const char *) big5, len); if (l < 0) + { + if (noError) + break; report_invalid_encoding(PG_BIG5, (const char *) big5, len); + } big5buf = (c1 << 8) | big5[1]; cnsBuf = BIG5toCNS(big5buf, &lc); if (lc != 0) @@ -412,20 +488,27 @@ big52mic(const unsigned char *big5, unsigned char *p, int len) *p++ = cnsBuf & 0x00ff; } else + { + if (noError) + break; report_untranslatable_char(PG_BIG5, PG_MULE_INTERNAL, (const char *) big5, len); + } big5 += l; len -= l; } *p = '\0'; + + return big5 - start; } /* * MIC ---> Big5 */ -static void -mic2big5(const unsigned char *mic, unsigned char *p, int len) +static int +mic2big5(const unsigned char *mic, unsigned char *p, int len, bool noError) { + const unsigned char *start = mic; unsigned short c1; unsigned short big5buf, cnsBuf; @@ -438,8 +521,12 @@ mic2big5(const unsigned char *mic, unsigned char *p, int len) { /* ASCII */ if (c1 == 0) + { + if (noError) + break; report_invalid_encoding(PG_MULE_INTERNAL, (const char *) mic, len); + } *p++ = c1; mic++; len--; @@ -447,8 +534,12 @@ mic2big5(const unsigned char *mic, unsigned char *p, int len) } l = pg_encoding_verifymbchar(PG_MULE_INTERNAL, (const char *) mic, len); if (l < 0) + { + if (noError) + break; report_invalid_encoding(PG_MULE_INTERNAL, (const char *) mic, len); + } if (c1 == LC_CNS11643_1 || c1 == LC_CNS11643_2 || c1 == LCPRV2_B) { if (c1 == LCPRV2_B) @@ -462,16 +553,26 @@ mic2big5(const unsigned char *mic, unsigned char *p, int len) } big5buf = CNStoBIG5(cnsBuf, c1); if (big5buf == 0) + { + if (noError) + break; report_untranslatable_char(PG_MULE_INTERNAL, PG_BIG5, (const char *) mic, len); + } *p++ = (big5buf >> 8) & 0x00ff; *p++ = big5buf & 0x00ff; } else + { + if (noError) + break; report_untranslatable_char(PG_MULE_INTERNAL, PG_BIG5, (const char *) mic, len); + } mic += l; len -= l; } *p = '\0'; + + return mic - start; } diff --git a/src/backend/utils/mb/conversion_procs/latin2_and_win1250/latin2_and_win1250.c b/src/backend/utils/mb/conversion_procs/latin2_and_win1250/latin2_and_win1250.c index 2e28e6780a5..8610fcb69aa 100644 --- a/src/backend/utils/mb/conversion_procs/latin2_and_win1250/latin2_and_win1250.c +++ b/src/backend/utils/mb/conversion_procs/latin2_and_win1250/latin2_and_win1250.c @@ -30,8 +30,11 @@ PG_FUNCTION_INFO_V1(win1250_to_latin2); * INTEGER, -- destination encoding id * CSTRING, -- source string (null terminated C string) * CSTRING, -- destination string (null terminated C string) - * INTEGER -- source string length - * ) returns VOID; + * INTEGER, -- source string length + * BOOL -- if true, don't throw an error if conversion fails + * ) returns INTEGER; + * + * Returns the number of bytes successfully converted. * ---------- */ @@ -82,12 +85,14 @@ latin2_to_mic(PG_FUNCTION_ARGS) unsigned char *src = (unsigned char *) PG_GETARG_CSTRING(2); unsigned char *dest = (unsigned char *) PG_GETARG_CSTRING(3); int len = PG_GETARG_INT32(4); + bool noError = PG_GETARG_BOOL(5); + int converted; CHECK_ENCODING_CONVERSION_ARGS(PG_LATIN2, PG_MULE_INTERNAL); - latin2mic(src, dest, len, LC_ISO8859_2, PG_LATIN2); + converted = latin2mic(src, dest, len, LC_ISO8859_2, PG_LATIN2, noError); - PG_RETURN_VOID(); + PG_RETURN_INT32(converted); } Datum @@ -96,12 +101,14 @@ mic_to_latin2(PG_FUNCTION_ARGS) unsigned char *src = (unsigned char *) PG_GETARG_CSTRING(2); unsigned char *dest = (unsigned char *) PG_GETARG_CSTRING(3); int len = PG_GETARG_INT32(4); + bool noError = PG_GETARG_BOOL(5); + int converted; CHECK_ENCODING_CONVERSION_ARGS(PG_MULE_INTERNAL, PG_LATIN2); - mic2latin(src, dest, len, LC_ISO8859_2, PG_LATIN2); + converted = mic2latin(src, dest, len, LC_ISO8859_2, PG_LATIN2, noError); - PG_RETURN_VOID(); + PG_RETURN_INT32(converted); } Datum @@ -110,13 +117,15 @@ win1250_to_mic(PG_FUNCTION_ARGS) unsigned char *src = (unsigned char *) PG_GETARG_CSTRING(2); unsigned char *dest = (unsigned char *) PG_GETARG_CSTRING(3); int len = PG_GETARG_INT32(4); + bool noError = PG_GETARG_BOOL(5); + int converted; CHECK_ENCODING_CONVERSION_ARGS(PG_WIN1250, PG_MULE_INTERNAL); - latin2mic_with_table(src, dest, len, LC_ISO8859_2, PG_WIN1250, - win1250_2_iso88592); + converted = latin2mic_with_table(src, dest, len, LC_ISO8859_2, PG_WIN1250, + win1250_2_iso88592, noError); - PG_RETURN_VOID(); + PG_RETURN_INT32(converted); } Datum @@ -125,13 +134,15 @@ mic_to_win1250(PG_FUNCTION_ARGS) unsigned char *src = (unsigned char *) PG_GETARG_CSTRING(2); unsigned char *dest = (unsigned char *) PG_GETARG_CSTRING(3); int len = PG_GETARG_INT32(4); + bool noError = PG_GETARG_BOOL(5); + int converted; CHECK_ENCODING_CONVERSION_ARGS(PG_MULE_INTERNAL, PG_WIN1250); - mic2latin_with_table(src, dest, len, LC_ISO8859_2, PG_WIN1250, - iso88592_2_win1250); + converted = mic2latin_with_table(src, dest, len, LC_ISO8859_2, PG_WIN1250, + iso88592_2_win1250, noError); - PG_RETURN_VOID(); + PG_RETURN_INT32(converted); } Datum @@ -140,12 +151,15 @@ latin2_to_win1250(PG_FUNCTION_ARGS) unsigned char *src = (unsigned char *) PG_GETARG_CSTRING(2); unsigned char *dest = (unsigned char *) PG_GETARG_CSTRING(3); int len = PG_GETARG_INT32(4); + bool noError = PG_GETARG_BOOL(5); + int converted; CHECK_ENCODING_CONVERSION_ARGS(PG_LATIN2, PG_WIN1250); - local2local(src, dest, len, PG_LATIN2, PG_WIN1250, iso88592_2_win1250); + converted = local2local(src, dest, len, PG_LATIN2, PG_WIN1250, + iso88592_2_win1250, noError); - PG_RETURN_VOID(); + PG_RETURN_INT32(converted); } Datum @@ -154,10 +168,13 @@ win1250_to_latin2(PG_FUNCTION_ARGS) unsigned char *src = (unsigned char *) PG_GETARG_CSTRING(2); unsigned char *dest = (unsigned char *) PG_GETARG_CSTRING(3); int len = PG_GETARG_INT32(4); + bool noError = PG_GETARG_BOOL(5); + int converted; CHECK_ENCODING_CONVERSION_ARGS(PG_WIN1250, PG_LATIN2); - local2local(src, dest, len, PG_WIN1250, PG_LATIN2, win1250_2_iso88592); + converted = local2local(src, dest, len, PG_WIN1250, PG_LATIN2, + win1250_2_iso88592, noError); - PG_RETURN_VOID(); + PG_RETURN_INT32(converted); } diff --git a/src/backend/utils/mb/conversion_procs/latin_and_mic/latin_and_mic.c b/src/backend/utils/mb/conversion_procs/latin_and_mic/latin_and_mic.c index bc651410f21..bff27d1c295 100644 --- a/src/backend/utils/mb/conversion_procs/latin_and_mic/latin_and_mic.c +++ b/src/backend/utils/mb/conversion_procs/latin_and_mic/latin_and_mic.c @@ -30,8 +30,11 @@ PG_FUNCTION_INFO_V1(mic_to_latin4); * INTEGER, -- destination encoding id * CSTRING, -- source string (null terminated C string) * CSTRING, -- destination string (null terminated C string) - * INTEGER -- source string length - * ) returns VOID; + * INTEGER, -- source string length + * BOOL -- if true, don't throw an error if conversion fails + * ) returns INTEGER; + * + * Returns the number of bytes successfully converted. * ---------- */ @@ -42,12 +45,14 @@ latin1_to_mic(PG_FUNCTION_ARGS) unsigned char *src = (unsigned char *) PG_GETARG_CSTRING(2); unsigned char *dest = (unsigned char *) PG_GETARG_CSTRING(3); int len = PG_GETARG_INT32(4); + bool noError = PG_GETARG_BOOL(5); + int converted; CHECK_ENCODING_CONVERSION_ARGS(PG_LATIN1, PG_MULE_INTERNAL); - latin2mic(src, dest, len, LC_ISO8859_1, PG_LATIN1); + converted = latin2mic(src, dest, len, LC_ISO8859_1, PG_LATIN1, noError); - PG_RETURN_VOID(); + PG_RETURN_INT32(converted); } Datum @@ -56,12 +61,14 @@ mic_to_latin1(PG_FUNCTION_ARGS) unsigned char *src = (unsigned char *) PG_GETARG_CSTRING(2); unsigned char *dest = (unsigned char *) PG_GETARG_CSTRING(3); int len = PG_GETARG_INT32(4); + bool noError = PG_GETARG_BOOL(5); + int converted; CHECK_ENCODING_CONVERSION_ARGS(PG_MULE_INTERNAL, PG_LATIN1); - mic2latin(src, dest, len, LC_ISO8859_1, PG_LATIN1); + converted = mic2latin(src, dest, len, LC_ISO8859_1, PG_LATIN1, noError); - PG_RETURN_VOID(); + PG_RETURN_INT32(converted); } Datum @@ -70,12 +77,14 @@ latin3_to_mic(PG_FUNCTION_ARGS) unsigned char *src = (unsigned char *) PG_GETARG_CSTRING(2); unsigned char *dest = (unsigned char *) PG_GETARG_CSTRING(3); int len = PG_GETARG_INT32(4); + bool noError = PG_GETARG_BOOL(5); + int converted; CHECK_ENCODING_CONVERSION_ARGS(PG_LATIN3, PG_MULE_INTERNAL); - latin2mic(src, dest, len, LC_ISO8859_3, PG_LATIN3); + converted = latin2mic(src, dest, len, LC_ISO8859_3, PG_LATIN3, noError); - PG_RETURN_VOID(); + PG_RETURN_INT32(converted); } Datum @@ -84,12 +93,14 @@ mic_to_latin3(PG_FUNCTION_ARGS) unsigned char *src = (unsigned char *) PG_GETARG_CSTRING(2); unsigned char *dest = (unsigned char *) PG_GETARG_CSTRING(3); int len = PG_GETARG_INT32(4); + bool noError = PG_GETARG_BOOL(5); + int converted; CHECK_ENCODING_CONVERSION_ARGS(PG_MULE_INTERNAL, PG_LATIN3); - mic2latin(src, dest, len, LC_ISO8859_3, PG_LATIN3); + converted = mic2latin(src, dest, len, LC_ISO8859_3, PG_LATIN3, noError); - PG_RETURN_VOID(); + PG_RETURN_INT32(converted); } Datum @@ -98,12 +109,14 @@ latin4_to_mic(PG_FUNCTION_ARGS) unsigned char *src = (unsigned char *) PG_GETARG_CSTRING(2); unsigned char *dest = (unsigned char *) PG_GETARG_CSTRING(3); int len = PG_GETARG_INT32(4); + bool noError = PG_GETARG_BOOL(5); + int converted; CHECK_ENCODING_CONVERSION_ARGS(PG_LATIN4, PG_MULE_INTERNAL); - latin2mic(src, dest, len, LC_ISO8859_4, PG_LATIN4); + converted = latin2mic(src, dest, len, LC_ISO8859_4, PG_LATIN4, noError); - PG_RETURN_VOID(); + PG_RETURN_INT32(converted); } Datum @@ -112,10 +125,12 @@ mic_to_latin4(PG_FUNCTION_ARGS) unsigned char *src = (unsigned char *) PG_GETARG_CSTRING(2); unsigned char *dest = (unsigned char *) PG_GETARG_CSTRING(3); int len = PG_GETARG_INT32(4); + bool noError = PG_GETARG_BOOL(5); + int converted; CHECK_ENCODING_CONVERSION_ARGS(PG_MULE_INTERNAL, PG_LATIN4); - mic2latin(src, dest, len, LC_ISO8859_4, PG_LATIN4); + converted = mic2latin(src, dest, len, LC_ISO8859_4, PG_LATIN4, noError); - PG_RETURN_VOID(); + PG_RETURN_INT32(converted); } diff --git a/src/backend/utils/mb/conversion_procs/utf8_and_big5/utf8_and_big5.c b/src/backend/utils/mb/conversion_procs/utf8_and_big5/utf8_and_big5.c index d6067cdc24e..3838b15cab9 100644 --- a/src/backend/utils/mb/conversion_procs/utf8_and_big5/utf8_and_big5.c +++ b/src/backend/utils/mb/conversion_procs/utf8_and_big5/utf8_and_big5.c @@ -28,8 +28,11 @@ PG_FUNCTION_INFO_V1(utf8_to_big5); * INTEGER, -- destination encoding id * CSTRING, -- source string (null terminated C string) * CSTRING, -- destination string (null terminated C string) - * INTEGER -- source string length - * ) returns VOID; + * INTEGER, -- source string length + * BOOL -- if true, don't throw an error if conversion fails + * ) returns INTEGER; + * + * Returns the number of bytes successfully converted. * ---------- */ Datum @@ -38,16 +41,19 @@ big5_to_utf8(PG_FUNCTION_ARGS) unsigned char *src = (unsigned char *) PG_GETARG_CSTRING(2); unsigned char *dest = (unsigned char *) PG_GETARG_CSTRING(3); int len = PG_GETARG_INT32(4); + bool noError = PG_GETARG_BOOL(5); + int converted; CHECK_ENCODING_CONVERSION_ARGS(PG_BIG5, PG_UTF8); - LocalToUtf(src, len, dest, - &big5_to_unicode_tree, - NULL, 0, - NULL, - PG_BIG5); + converted = LocalToUtf(src, len, dest, + &big5_to_unicode_tree, + NULL, 0, + NULL, + PG_BIG5, + noError); - PG_RETURN_VOID(); + PG_RETURN_INT32(converted); } Datum @@ -56,14 +62,17 @@ utf8_to_big5(PG_FUNCTION_ARGS) unsigned char *src = (unsigned char *) PG_GETARG_CSTRING(2); unsigned char *dest = (unsigned char *) PG_GETARG_CSTRING(3); int len = PG_GETARG_INT32(4); + bool noError = PG_GETARG_BOOL(5); + int converted; CHECK_ENCODING_CONVERSION_ARGS(PG_UTF8, PG_BIG5); - UtfToLocal(src, len, dest, - &big5_from_unicode_tree, - NULL, 0, - NULL, - PG_BIG5); + converted = UtfToLocal(src, len, dest, + &big5_from_unicode_tree, + NULL, 0, + NULL, + PG_BIG5, + noError); - PG_RETURN_VOID(); + PG_RETURN_INT32(converted); } diff --git a/src/backend/utils/mb/conversion_procs/utf8_and_cyrillic/utf8_and_cyrillic.c b/src/backend/utils/mb/conversion_procs/utf8_and_cyrillic/utf8_and_cyrillic.c index ed90e8e682e..75719fe5f1b 100644 --- a/src/backend/utils/mb/conversion_procs/utf8_and_cyrillic/utf8_and_cyrillic.c +++ b/src/backend/utils/mb/conversion_procs/utf8_and_cyrillic/utf8_and_cyrillic.c @@ -33,8 +33,11 @@ PG_FUNCTION_INFO_V1(koi8u_to_utf8); * INTEGER, -- destination encoding id * CSTRING, -- source string (null terminated C string) * CSTRING, -- destination string (null terminated C string) - * INTEGER -- source string length - * ) returns VOID; + * INTEGER, -- source string length + * BOOL -- if true, don't throw an error if conversion fails + * ) returns INTEGER; + * + * Returns the number of bytes successfully converted. * ---------- */ @@ -44,16 +47,19 @@ utf8_to_koi8r(PG_FUNCTION_ARGS) unsigned char *src = (unsigned char *) PG_GETARG_CSTRING(2); unsigned char *dest = (unsigned char *) PG_GETARG_CSTRING(3); int len = PG_GETARG_INT32(4); + bool noError = PG_GETARG_BOOL(5); + int converted; CHECK_ENCODING_CONVERSION_ARGS(PG_UTF8, PG_KOI8R); - UtfToLocal(src, len, dest, - &koi8r_from_unicode_tree, - NULL, 0, - NULL, - PG_KOI8R); + converted = UtfToLocal(src, len, dest, + &koi8r_from_unicode_tree, + NULL, 0, + NULL, + PG_KOI8R, + noError); - PG_RETURN_VOID(); + PG_RETURN_INT32(converted); } Datum @@ -62,16 +68,19 @@ koi8r_to_utf8(PG_FUNCTION_ARGS) unsigned char *src = (unsigned char *) PG_GETARG_CSTRING(2); unsigned char *dest = (unsigned char *) PG_GETARG_CSTRING(3); int len = PG_GETARG_INT32(4); + bool noError = PG_GETARG_BOOL(5); + int converted; CHECK_ENCODING_CONVERSION_ARGS(PG_KOI8R, PG_UTF8); - LocalToUtf(src, len, dest, - &koi8r_to_unicode_tree, - NULL, 0, - NULL, - PG_KOI8R); + converted = LocalToUtf(src, len, dest, + &koi8r_to_unicode_tree, + NULL, 0, + NULL, + PG_KOI8R, + noError); - PG_RETURN_VOID(); + PG_RETURN_INT32(converted); } Datum @@ -80,16 +89,19 @@ utf8_to_koi8u(PG_FUNCTION_ARGS) unsigned char *src = (unsigned char *) PG_GETARG_CSTRING(2); unsigned char *dest = (unsigned char *) PG_GETARG_CSTRING(3); int len = PG_GETARG_INT32(4); + bool noError = PG_GETARG_BOOL(5); + int converted; CHECK_ENCODING_CONVERSION_ARGS(PG_UTF8, PG_KOI8U); - UtfToLocal(src, len, dest, - &koi8u_from_unicode_tree, - NULL, 0, - NULL, - PG_KOI8U); + converted = UtfToLocal(src, len, dest, + &koi8u_from_unicode_tree, + NULL, 0, + NULL, + PG_KOI8U, + noError); - PG_RETURN_VOID(); + PG_RETURN_INT32(converted); } Datum @@ -98,14 +110,17 @@ koi8u_to_utf8(PG_FUNCTION_ARGS) unsigned char *src = (unsigned char *) PG_GETARG_CSTRING(2); unsigned char *dest = (unsigned char *) PG_GETARG_CSTRING(3); int len = PG_GETARG_INT32(4); + bool noError = PG_GETARG_BOOL(5); + int converted; CHECK_ENCODING_CONVERSION_ARGS(PG_KOI8U, PG_UTF8); - LocalToUtf(src, len, dest, - &koi8u_to_unicode_tree, - NULL, 0, - NULL, - PG_KOI8U); + converted = LocalToUtf(src, len, dest, + &koi8u_to_unicode_tree, + NULL, 0, + NULL, + PG_KOI8U, + noError); - PG_RETURN_VOID(); + PG_RETURN_INT32(converted); } diff --git a/src/backend/utils/mb/conversion_procs/utf8_and_euc2004/utf8_and_euc2004.c b/src/backend/utils/mb/conversion_procs/utf8_and_euc2004/utf8_and_euc2004.c index d699affce47..5391001951a 100644 --- a/src/backend/utils/mb/conversion_procs/utf8_and_euc2004/utf8_and_euc2004.c +++ b/src/backend/utils/mb/conversion_procs/utf8_and_euc2004/utf8_and_euc2004.c @@ -28,8 +28,11 @@ PG_FUNCTION_INFO_V1(utf8_to_euc_jis_2004); * INTEGER, -- destination encoding id * CSTRING, -- source string (null terminated C string) * CSTRING, -- destination string (null terminated C string) - * INTEGER -- source string length - * ) returns VOID; + * INTEGER, -- source string length + * BOOL -- if true, don't throw an error if conversion fails + * ) returns INTEGER; + * + * Returns the number of bytes successfully converted. * ---------- */ Datum @@ -38,16 +41,19 @@ euc_jis_2004_to_utf8(PG_FUNCTION_ARGS) unsigned char *src = (unsigned char *) PG_GETARG_CSTRING(2); unsigned char *dest = (unsigned char *) PG_GETARG_CSTRING(3); int len = PG_GETARG_INT32(4); + bool noError = PG_GETARG_BOOL(5); + int converted; CHECK_ENCODING_CONVERSION_ARGS(PG_EUC_JIS_2004, PG_UTF8); - LocalToUtf(src, len, dest, - &euc_jis_2004_to_unicode_tree, - LUmapEUC_JIS_2004_combined, lengthof(LUmapEUC_JIS_2004_combined), - NULL, - PG_EUC_JIS_2004); + converted = LocalToUtf(src, len, dest, + &euc_jis_2004_to_unicode_tree, + LUmapEUC_JIS_2004_combined, lengthof(LUmapEUC_JIS_2004_combined), + NULL, + PG_EUC_JIS_2004, + noError); - PG_RETURN_VOID(); + PG_RETURN_INT32(converted); } Datum @@ -56,14 +62,17 @@ utf8_to_euc_jis_2004(PG_FUNCTION_ARGS) unsigned char *src = (unsigned char *) PG_GETARG_CSTRING(2); unsigned char *dest = (unsigned char *) PG_GETARG_CSTRING(3); int len = PG_GETARG_INT32(4); + bool noError = PG_GETARG_BOOL(5); + int converted; CHECK_ENCODING_CONVERSION_ARGS(PG_UTF8, PG_EUC_JIS_2004); - UtfToLocal(src, len, dest, - &euc_jis_2004_from_unicode_tree, - ULmapEUC_JIS_2004_combined, lengthof(ULmapEUC_JIS_2004_combined), - NULL, - PG_EUC_JIS_2004); + converted = UtfToLocal(src, len, dest, + &euc_jis_2004_from_unicode_tree, + ULmapEUC_JIS_2004_combined, lengthof(ULmapEUC_JIS_2004_combined), + NULL, + PG_EUC_JIS_2004, + noError); - PG_RETURN_VOID(); + PG_RETURN_INT32(converted); } diff --git a/src/backend/utils/mb/conversion_procs/utf8_and_euc_cn/utf8_and_euc_cn.c b/src/backend/utils/mb/conversion_procs/utf8_and_euc_cn/utf8_and_euc_cn.c index d7c0ba6a58b..c87d1bf2398 100644 --- a/src/backend/utils/mb/conversion_procs/utf8_and_euc_cn/utf8_and_euc_cn.c +++ b/src/backend/utils/mb/conversion_procs/utf8_and_euc_cn/utf8_and_euc_cn.c @@ -28,8 +28,11 @@ PG_FUNCTION_INFO_V1(utf8_to_euc_cn); * INTEGER, -- destination encoding id * CSTRING, -- source string (null terminated C string) * CSTRING, -- destination string (null terminated C string) - * INTEGER -- source string length - * ) returns VOID; + * INTEGER, -- source string length + * BOOL -- if true, don't throw an error if conversion fails + * ) returns INTEGER; + * + * Returns the number of bytes successfully converted. * ---------- */ Datum @@ -38,16 +41,19 @@ euc_cn_to_utf8(PG_FUNCTION_ARGS) unsigned char *src = (unsigned char *) PG_GETARG_CSTRING(2); unsigned char *dest = (unsigned char *) PG_GETARG_CSTRING(3); int len = PG_GETARG_INT32(4); + bool noError = PG_GETARG_BOOL(5); + int converted; CHECK_ENCODING_CONVERSION_ARGS(PG_EUC_CN, PG_UTF8); - LocalToUtf(src, len, dest, - &euc_cn_to_unicode_tree, - NULL, 0, - NULL, - PG_EUC_CN); + converted = LocalToUtf(src, len, dest, + &euc_cn_to_unicode_tree, + NULL, 0, + NULL, + PG_EUC_CN, + noError); - PG_RETURN_VOID(); + PG_RETURN_INT32(converted); } Datum @@ -56,14 +62,17 @@ utf8_to_euc_cn(PG_FUNCTION_ARGS) unsigned char *src = (unsigned char *) PG_GETARG_CSTRING(2); unsigned char *dest = (unsigned char *) PG_GETARG_CSTRING(3); int len = PG_GETARG_INT32(4); + bool noError = PG_GETARG_BOOL(5); + int converted; CHECK_ENCODING_CONVERSION_ARGS(PG_UTF8, PG_EUC_CN); - UtfToLocal(src, len, dest, - &euc_cn_from_unicode_tree, - NULL, 0, - NULL, - PG_EUC_CN); + converted = UtfToLocal(src, len, dest, + &euc_cn_from_unicode_tree, + NULL, 0, + NULL, + PG_EUC_CN, + noError); - PG_RETURN_VOID(); + PG_RETURN_INT32(converted); } diff --git a/src/backend/utils/mb/conversion_procs/utf8_and_euc_jp/utf8_and_euc_jp.c b/src/backend/utils/mb/conversion_procs/utf8_and_euc_jp/utf8_and_euc_jp.c index 13a3a23e77b..6a55134db21 100644 --- a/src/backend/utils/mb/conversion_procs/utf8_and_euc_jp/utf8_and_euc_jp.c +++ b/src/backend/utils/mb/conversion_procs/utf8_and_euc_jp/utf8_and_euc_jp.c @@ -28,8 +28,11 @@ PG_FUNCTION_INFO_V1(utf8_to_euc_jp); * INTEGER, -- destination encoding id * CSTRING, -- source string (null terminated C string) * CSTRING, -- destination string (null terminated C string) - * INTEGER -- source string length - * ) returns VOID; + * INTEGER, -- source string length + * BOOL -- if true, don't throw an error if conversion fails + * ) returns INTEGER; + * + * Returns the number of bytes successfully converted. * ---------- */ Datum @@ -38,16 +41,19 @@ euc_jp_to_utf8(PG_FUNCTION_ARGS) unsigned char *src = (unsigned char *) PG_GETARG_CSTRING(2); unsigned char *dest = (unsigned char *) PG_GETARG_CSTRING(3); int len = PG_GETARG_INT32(4); + bool noError = PG_GETARG_BOOL(5); + int converted; CHECK_ENCODING_CONVERSION_ARGS(PG_EUC_JP, PG_UTF8); - LocalToUtf(src, len, dest, - &euc_jp_to_unicode_tree, - NULL, 0, - NULL, - PG_EUC_JP); + converted = LocalToUtf(src, len, dest, + &euc_jp_to_unicode_tree, + NULL, 0, + NULL, + PG_EUC_JP, + noError); - PG_RETURN_VOID(); + PG_RETURN_INT32(converted); } Datum @@ -56,14 +62,17 @@ utf8_to_euc_jp(PG_FUNCTION_ARGS) unsigned char *src = (unsigned char *) PG_GETARG_CSTRING(2); unsigned char *dest = (unsigned char *) PG_GETARG_CSTRING(3); int len = PG_GETARG_INT32(4); + bool noError = PG_GETARG_BOOL(5); + int converted; CHECK_ENCODING_CONVERSION_ARGS(PG_UTF8, PG_EUC_JP); - UtfToLocal(src, len, dest, - &euc_jp_from_unicode_tree, - NULL, 0, - NULL, - PG_EUC_JP); + converted = UtfToLocal(src, len, dest, + &euc_jp_from_unicode_tree, + NULL, 0, + NULL, + PG_EUC_JP, + noError); - PG_RETURN_VOID(); + PG_RETURN_INT32(converted); } diff --git a/src/backend/utils/mb/conversion_procs/utf8_and_euc_kr/utf8_and_euc_kr.c b/src/backend/utils/mb/conversion_procs/utf8_and_euc_kr/utf8_and_euc_kr.c index 1bbb8aaef7b..fe1924e2fec 100644 --- a/src/backend/utils/mb/conversion_procs/utf8_and_euc_kr/utf8_and_euc_kr.c +++ b/src/backend/utils/mb/conversion_procs/utf8_and_euc_kr/utf8_and_euc_kr.c @@ -28,8 +28,11 @@ PG_FUNCTION_INFO_V1(utf8_to_euc_kr); * INTEGER, -- destination encoding id * CSTRING, -- source string (null terminated C string) * CSTRING, -- destination string (null terminated C string) - * INTEGER -- source string length - * ) returns VOID; + * INTEGER, -- source string length + * BOOL -- if true, don't throw an error if conversion fails + * ) returns INTEGER; + * + * Returns the number of bytes successfully converted. * ---------- */ Datum @@ -38,16 +41,19 @@ euc_kr_to_utf8(PG_FUNCTION_ARGS) unsigned char *src = (unsigned char *) PG_GETARG_CSTRING(2); unsigned char *dest = (unsigned char *) PG_GETARG_CSTRING(3); int len = PG_GETARG_INT32(4); + bool noError = PG_GETARG_BOOL(5); + int converted; CHECK_ENCODING_CONVERSION_ARGS(PG_EUC_KR, PG_UTF8); - LocalToUtf(src, len, dest, - &euc_kr_to_unicode_tree, - NULL, 0, - NULL, - PG_EUC_KR); + converted = LocalToUtf(src, len, dest, + &euc_kr_to_unicode_tree, + NULL, 0, + NULL, + PG_EUC_KR, + noError); - PG_RETURN_VOID(); + PG_RETURN_INT32(converted); } Datum @@ -56,14 +62,17 @@ utf8_to_euc_kr(PG_FUNCTION_ARGS) unsigned char *src = (unsigned char *) PG_GETARG_CSTRING(2); unsigned char *dest = (unsigned char *) PG_GETARG_CSTRING(3); int len = PG_GETARG_INT32(4); + bool noError = PG_GETARG_BOOL(5); + int converted; CHECK_ENCODING_CONVERSION_ARGS(PG_UTF8, PG_EUC_KR); - UtfToLocal(src, len, dest, - &euc_kr_from_unicode_tree, - NULL, 0, - NULL, - PG_EUC_KR); + converted = UtfToLocal(src, len, dest, + &euc_kr_from_unicode_tree, + NULL, 0, + NULL, + PG_EUC_KR, + noError); - PG_RETURN_VOID(); + PG_RETURN_INT32(converted); } diff --git a/src/backend/utils/mb/conversion_procs/utf8_and_euc_tw/utf8_and_euc_tw.c b/src/backend/utils/mb/conversion_procs/utf8_and_euc_tw/utf8_and_euc_tw.c index 9830045dccd..68215659b57 100644 --- a/src/backend/utils/mb/conversion_procs/utf8_and_euc_tw/utf8_and_euc_tw.c +++ b/src/backend/utils/mb/conversion_procs/utf8_and_euc_tw/utf8_and_euc_tw.c @@ -28,8 +28,11 @@ PG_FUNCTION_INFO_V1(utf8_to_euc_tw); * INTEGER, -- destination encoding id * CSTRING, -- source string (null terminated C string) * CSTRING, -- destination string (null terminated C string) - * INTEGER -- source string length - * ) returns VOID; + * INTEGER, -- source string length + * BOOL -- if true, don't throw an error if conversion fails + * ) returns INTEGER; + * + * Returns the number of bytes successfully converted. * ---------- */ Datum @@ -38,16 +41,19 @@ euc_tw_to_utf8(PG_FUNCTION_ARGS) unsigned char *src = (unsigned char *) PG_GETARG_CSTRING(2); unsigned char *dest = (unsigned char *) PG_GETARG_CSTRING(3); int len = PG_GETARG_INT32(4); + bool noError = PG_GETARG_BOOL(5); + int converted; CHECK_ENCODING_CONVERSION_ARGS(PG_EUC_TW, PG_UTF8); - LocalToUtf(src, len, dest, - &euc_tw_to_unicode_tree, - NULL, 0, - NULL, - PG_EUC_TW); + converted = LocalToUtf(src, len, dest, + &euc_tw_to_unicode_tree, + NULL, 0, + NULL, + PG_EUC_TW, + noError); - PG_RETURN_VOID(); + PG_RETURN_INT32(converted); } Datum @@ -56,14 +62,17 @@ utf8_to_euc_tw(PG_FUNCTION_ARGS) unsigned char *src = (unsigned char *) PG_GETARG_CSTRING(2); unsigned char *dest = (unsigned char *) PG_GETARG_CSTRING(3); int len = PG_GETARG_INT32(4); + bool noError = PG_GETARG_BOOL(5); + int converted; CHECK_ENCODING_CONVERSION_ARGS(PG_UTF8, PG_EUC_TW); - UtfToLocal(src, len, dest, - &euc_tw_from_unicode_tree, - NULL, 0, - NULL, - PG_EUC_TW); + converted = UtfToLocal(src, len, dest, + &euc_tw_from_unicode_tree, + NULL, 0, + NULL, + PG_EUC_TW, + noError); - PG_RETURN_VOID(); + PG_RETURN_INT32(converted); } diff --git a/src/backend/utils/mb/conversion_procs/utf8_and_gb18030/utf8_and_gb18030.c b/src/backend/utils/mb/conversion_procs/utf8_and_gb18030/utf8_and_gb18030.c index f86ecf27424..e1a59c39a4d 100644 --- a/src/backend/utils/mb/conversion_procs/utf8_and_gb18030/utf8_and_gb18030.c +++ b/src/backend/utils/mb/conversion_procs/utf8_and_gb18030/utf8_and_gb18030.c @@ -183,8 +183,11 @@ conv_utf8_to_18030(uint32 code) * INTEGER, -- destination encoding id * CSTRING, -- source string (null terminated C string) * CSTRING, -- destination string (null terminated C string) - * INTEGER -- source string length - * ) returns VOID; + * INTEGER, -- source string length + * BOOL -- if true, don't throw an error if conversion fails + * ) returns INTEGER; + * + * Returns the number of bytes successfully converted. * ---------- */ Datum @@ -193,16 +196,19 @@ gb18030_to_utf8(PG_FUNCTION_ARGS) unsigned char *src = (unsigned char *) PG_GETARG_CSTRING(2); unsigned char *dest = (unsigned char *) PG_GETARG_CSTRING(3); int len = PG_GETARG_INT32(4); + bool noError = PG_GETARG_BOOL(5); + int converted; CHECK_ENCODING_CONVERSION_ARGS(PG_GB18030, PG_UTF8); - LocalToUtf(src, len, dest, - &gb18030_to_unicode_tree, - NULL, 0, - conv_18030_to_utf8, - PG_GB18030); + converted = LocalToUtf(src, len, dest, + &gb18030_to_unicode_tree, + NULL, 0, + conv_18030_to_utf8, + PG_GB18030, + noError); - PG_RETURN_VOID(); + PG_RETURN_INT32(converted); } Datum @@ -211,14 +217,17 @@ utf8_to_gb18030(PG_FUNCTION_ARGS) unsigned char *src = (unsigned char *) PG_GETARG_CSTRING(2); unsigned char *dest = (unsigned char *) PG_GETARG_CSTRING(3); int len = PG_GETARG_INT32(4); + bool noError = PG_GETARG_BOOL(5); + int converted; CHECK_ENCODING_CONVERSION_ARGS(PG_UTF8, PG_GB18030); - UtfToLocal(src, len, dest, - &gb18030_from_unicode_tree, - NULL, 0, - conv_utf8_to_18030, - PG_GB18030); + converted = UtfToLocal(src, len, dest, + &gb18030_from_unicode_tree, + NULL, 0, + conv_utf8_to_18030, + PG_GB18030, + noError); - PG_RETURN_VOID(); + PG_RETURN_INT32(converted); } diff --git a/src/backend/utils/mb/conversion_procs/utf8_and_gbk/utf8_and_gbk.c b/src/backend/utils/mb/conversion_procs/utf8_and_gbk/utf8_and_gbk.c index 2ab8b16c8a8..881386d5347 100644 --- a/src/backend/utils/mb/conversion_procs/utf8_and_gbk/utf8_and_gbk.c +++ b/src/backend/utils/mb/conversion_procs/utf8_and_gbk/utf8_and_gbk.c @@ -28,8 +28,11 @@ PG_FUNCTION_INFO_V1(utf8_to_gbk); * INTEGER, -- destination encoding id * CSTRING, -- source string (null terminated C string) * CSTRING, -- destination string (null terminated C string) - * INTEGER -- source string length - * ) returns VOID; + * INTEGER, -- source string length + * BOOL -- if true, don't throw an error if conversion fails + * ) returns INTEGER; + * + * Returns the number of bytes successfully converted. * ---------- */ Datum @@ -38,16 +41,19 @@ gbk_to_utf8(PG_FUNCTION_ARGS) unsigned char *src = (unsigned char *) PG_GETARG_CSTRING(2); unsigned char *dest = (unsigned char *) PG_GETARG_CSTRING(3); int len = PG_GETARG_INT32(4); + bool noError = PG_GETARG_BOOL(5); + int converted; CHECK_ENCODING_CONVERSION_ARGS(PG_GBK, PG_UTF8); - LocalToUtf(src, len, dest, - &gbk_to_unicode_tree, - NULL, 0, - NULL, - PG_GBK); + converted = LocalToUtf(src, len, dest, + &gbk_to_unicode_tree, + NULL, 0, + NULL, + PG_GBK, + noError); - PG_RETURN_VOID(); + PG_RETURN_INT32(converted); } Datum @@ -56,14 +62,17 @@ utf8_to_gbk(PG_FUNCTION_ARGS) unsigned char *src = (unsigned char *) PG_GETARG_CSTRING(2); unsigned char *dest = (unsigned char *) PG_GETARG_CSTRING(3); int len = PG_GETARG_INT32(4); + bool noError = PG_GETARG_BOOL(5); + int converted; CHECK_ENCODING_CONVERSION_ARGS(PG_UTF8, PG_GBK); - UtfToLocal(src, len, dest, - &gbk_from_unicode_tree, - NULL, 0, - NULL, - PG_GBK); + converted = UtfToLocal(src, len, dest, + &gbk_from_unicode_tree, + NULL, 0, + NULL, + PG_GBK, + noError); - PG_RETURN_VOID(); + PG_RETURN_INT32(converted); } diff --git a/src/backend/utils/mb/conversion_procs/utf8_and_iso8859/utf8_and_iso8859.c b/src/backend/utils/mb/conversion_procs/utf8_and_iso8859/utf8_and_iso8859.c index 3e49f67ea2f..d93a521badf 100644 --- a/src/backend/utils/mb/conversion_procs/utf8_and_iso8859/utf8_and_iso8859.c +++ b/src/backend/utils/mb/conversion_procs/utf8_and_iso8859/utf8_and_iso8859.c @@ -52,8 +52,11 @@ PG_FUNCTION_INFO_V1(utf8_to_iso8859); * INTEGER, -- destination encoding id * CSTRING, -- source string (null terminated C string) * CSTRING, -- destination string (null terminated C string) - * INTEGER -- source string length - * ) returns VOID; + * INTEGER, -- source string length + * BOOL -- if true, don't throw an error if conversion fails + * ) returns INTEGER; + * + * Returns the number of bytes successfully converted. * ---------- */ @@ -100,6 +103,7 @@ iso8859_to_utf8(PG_FUNCTION_ARGS) unsigned char *src = (unsigned char *) PG_GETARG_CSTRING(2); unsigned char *dest = (unsigned char *) PG_GETARG_CSTRING(3); int len = PG_GETARG_INT32(4); + bool noError = PG_GETARG_BOOL(5); int i; CHECK_ENCODING_CONVERSION_ARGS(-1, PG_UTF8); @@ -108,12 +112,15 @@ iso8859_to_utf8(PG_FUNCTION_ARGS) { if (encoding == maps[i].encoding) { - LocalToUtf(src, len, dest, - maps[i].map1, - NULL, 0, - NULL, - encoding); - PG_RETURN_VOID(); + int converted; + + converted = LocalToUtf(src, len, dest, + maps[i].map1, + NULL, 0, + NULL, + encoding, + noError); + PG_RETURN_INT32(converted); } } @@ -122,7 +129,7 @@ iso8859_to_utf8(PG_FUNCTION_ARGS) errmsg("unexpected encoding ID %d for ISO 8859 character sets", encoding))); - PG_RETURN_VOID(); + PG_RETURN_INT32(0); } Datum @@ -132,6 +139,7 @@ utf8_to_iso8859(PG_FUNCTION_ARGS) unsigned char *src = (unsigned char *) PG_GETARG_CSTRING(2); unsigned char *dest = (unsigned char *) PG_GETARG_CSTRING(3); int len = PG_GETARG_INT32(4); + bool noError = PG_GETARG_BOOL(5); int i; CHECK_ENCODING_CONVERSION_ARGS(PG_UTF8, -1); @@ -140,12 +148,15 @@ utf8_to_iso8859(PG_FUNCTION_ARGS) { if (encoding == maps[i].encoding) { - UtfToLocal(src, len, dest, - maps[i].map2, - NULL, 0, - NULL, - encoding); - PG_RETURN_VOID(); + int converted; + + converted = UtfToLocal(src, len, dest, + maps[i].map2, + NULL, 0, + NULL, + encoding, + noError); + PG_RETURN_INT32(converted); } } @@ -154,5 +165,5 @@ utf8_to_iso8859(PG_FUNCTION_ARGS) errmsg("unexpected encoding ID %d for ISO 8859 character sets", encoding))); - PG_RETURN_VOID(); + PG_RETURN_INT32(0); } diff --git a/src/backend/utils/mb/conversion_procs/utf8_and_iso8859_1/utf8_and_iso8859_1.c b/src/backend/utils/mb/conversion_procs/utf8_and_iso8859_1/utf8_and_iso8859_1.c index 67e713cca11..d0dc4cca378 100644 --- a/src/backend/utils/mb/conversion_procs/utf8_and_iso8859_1/utf8_and_iso8859_1.c +++ b/src/backend/utils/mb/conversion_procs/utf8_and_iso8859_1/utf8_and_iso8859_1.c @@ -26,8 +26,11 @@ PG_FUNCTION_INFO_V1(utf8_to_iso8859_1); * INTEGER, -- destination encoding id * CSTRING, -- source string (null terminated C string) * CSTRING, -- destination string (null terminated C string) - * INTEGER -- source string length - * ) returns VOID; + * INTEGER, -- source string length + * BOOL -- if true, don't throw an error if conversion fails + * ) returns INTEGER; + * + * Returns the number of bytes successfully converted. * ---------- */ @@ -37,6 +40,8 @@ iso8859_1_to_utf8(PG_FUNCTION_ARGS) unsigned char *src = (unsigned char *) PG_GETARG_CSTRING(2); unsigned char *dest = (unsigned char *) PG_GETARG_CSTRING(3); int len = PG_GETARG_INT32(4); + bool noError = PG_GETARG_BOOL(5); + unsigned char *start = src; unsigned short c; CHECK_ENCODING_CONVERSION_ARGS(PG_LATIN1, PG_UTF8); @@ -45,7 +50,11 @@ iso8859_1_to_utf8(PG_FUNCTION_ARGS) { c = *src; if (c == 0) + { + if (noError) + break; report_invalid_encoding(PG_LATIN1, (const char *) src, len); + } if (!IS_HIGHBIT_SET(c)) *dest++ = c; else @@ -58,7 +67,7 @@ iso8859_1_to_utf8(PG_FUNCTION_ARGS) } *dest = '\0'; - PG_RETURN_VOID(); + PG_RETURN_INT32(src - start); } Datum @@ -67,6 +76,8 @@ utf8_to_iso8859_1(PG_FUNCTION_ARGS) unsigned char *src = (unsigned char *) PG_GETARG_CSTRING(2); unsigned char *dest = (unsigned char *) PG_GETARG_CSTRING(3); int len = PG_GETARG_INT32(4); + bool noError = PG_GETARG_BOOL(5); + unsigned char *start = src; unsigned short c, c1; @@ -76,7 +87,11 @@ utf8_to_iso8859_1(PG_FUNCTION_ARGS) { c = *src; if (c == 0) + { + if (noError) + break; report_invalid_encoding(PG_UTF8, (const char *) src, len); + } /* fast path for ASCII-subset characters */ if (!IS_HIGHBIT_SET(c)) { @@ -89,10 +104,18 @@ utf8_to_iso8859_1(PG_FUNCTION_ARGS) int l = pg_utf_mblen(src); if (l > len || !pg_utf8_islegal(src, l)) + { + if (noError) + break; report_invalid_encoding(PG_UTF8, (const char *) src, len); + } if (l != 2) + { + if (noError) + break; report_untranslatable_char(PG_UTF8, PG_LATIN1, (const char *) src, len); + } c1 = src[1] & 0x3f; c = ((c & 0x1f) << 6) | c1; if (c >= 0x80 && c <= 0xff) @@ -102,11 +125,15 @@ utf8_to_iso8859_1(PG_FUNCTION_ARGS) len -= 2; } else + { + if (noError) + break; report_untranslatable_char(PG_UTF8, PG_LATIN1, (const char *) src, len); + } } } *dest = '\0'; - PG_RETURN_VOID(); + PG_RETURN_INT32(src - start); } diff --git a/src/backend/utils/mb/conversion_procs/utf8_and_johab/utf8_and_johab.c b/src/backend/utils/mb/conversion_procs/utf8_and_johab/utf8_and_johab.c index 578f5df4e7f..317daa2d5ee 100644 --- a/src/backend/utils/mb/conversion_procs/utf8_and_johab/utf8_and_johab.c +++ b/src/backend/utils/mb/conversion_procs/utf8_and_johab/utf8_and_johab.c @@ -28,8 +28,11 @@ PG_FUNCTION_INFO_V1(utf8_to_johab); * INTEGER, -- destination encoding id * CSTRING, -- source string (null terminated C string) * CSTRING, -- destination string (null terminated C string) - * INTEGER -- source string length - * ) returns VOID; + * INTEGER, -- source string length + * BOOL -- if true, don't throw an error if conversion fails + * ) returns INTEGER; + * + * Returns the number of bytes successfully converted. * ---------- */ Datum @@ -38,16 +41,19 @@ johab_to_utf8(PG_FUNCTION_ARGS) unsigned char *src = (unsigned char *) PG_GETARG_CSTRING(2); unsigned char *dest = (unsigned char *) PG_GETARG_CSTRING(3); int len = PG_GETARG_INT32(4); + bool noError = PG_GETARG_BOOL(5); + int converted; CHECK_ENCODING_CONVERSION_ARGS(PG_JOHAB, PG_UTF8); - LocalToUtf(src, len, dest, - &johab_to_unicode_tree, - NULL, 0, - NULL, - PG_JOHAB); + converted = LocalToUtf(src, len, dest, + &johab_to_unicode_tree, + NULL, 0, + NULL, + PG_JOHAB, + noError); - PG_RETURN_VOID(); + PG_RETURN_INT32(converted); } Datum @@ -56,14 +62,17 @@ utf8_to_johab(PG_FUNCTION_ARGS) unsigned char *src = (unsigned char *) PG_GETARG_CSTRING(2); unsigned char *dest = (unsigned char *) PG_GETARG_CSTRING(3); int len = PG_GETARG_INT32(4); + bool noError = PG_GETARG_BOOL(5); + int converted; CHECK_ENCODING_CONVERSION_ARGS(PG_UTF8, PG_JOHAB); - UtfToLocal(src, len, dest, - &johab_from_unicode_tree, - NULL, 0, - NULL, - PG_JOHAB); + converted = UtfToLocal(src, len, dest, + &johab_from_unicode_tree, + NULL, 0, + NULL, + PG_JOHAB, + noError); - PG_RETURN_VOID(); + PG_RETURN_INT32(converted); } diff --git a/src/backend/utils/mb/conversion_procs/utf8_and_sjis/utf8_and_sjis.c b/src/backend/utils/mb/conversion_procs/utf8_and_sjis/utf8_and_sjis.c index dd9fc2975ad..4c9348aba59 100644 --- a/src/backend/utils/mb/conversion_procs/utf8_and_sjis/utf8_and_sjis.c +++ b/src/backend/utils/mb/conversion_procs/utf8_and_sjis/utf8_and_sjis.c @@ -28,8 +28,11 @@ PG_FUNCTION_INFO_V1(utf8_to_sjis); * INTEGER, -- destination encoding id * CSTRING, -- source string (null terminated C string) * CSTRING, -- destination string (null terminated C string) - * INTEGER -- source string length - * ) returns VOID; + * INTEGER, -- source string length + * BOOL -- if true, don't throw an error if conversion fails + * ) returns INTEGER; + * + * Returns the number of bytes successfully converted. * ---------- */ Datum @@ -38,16 +41,19 @@ sjis_to_utf8(PG_FUNCTION_ARGS) unsigned char *src = (unsigned char *) PG_GETARG_CSTRING(2); unsigned char *dest = (unsigned char *) PG_GETARG_CSTRING(3); int len = PG_GETARG_INT32(4); + bool noError = PG_GETARG_BOOL(5); + int converted; CHECK_ENCODING_CONVERSION_ARGS(PG_SJIS, PG_UTF8); - LocalToUtf(src, len, dest, - &sjis_to_unicode_tree, - NULL, 0, - NULL, - PG_SJIS); + converted = LocalToUtf(src, len, dest, + &sjis_to_unicode_tree, + NULL, 0, + NULL, + PG_SJIS, + noError); - PG_RETURN_VOID(); + PG_RETURN_INT32(converted); } Datum @@ -56,14 +62,17 @@ utf8_to_sjis(PG_FUNCTION_ARGS) unsigned char *src = (unsigned char *) PG_GETARG_CSTRING(2); unsigned char *dest = (unsigned char *) PG_GETARG_CSTRING(3); int len = PG_GETARG_INT32(4); + bool noError = PG_GETARG_BOOL(5); + int converted; CHECK_ENCODING_CONVERSION_ARGS(PG_UTF8, PG_SJIS); - UtfToLocal(src, len, dest, - &sjis_from_unicode_tree, - NULL, 0, - NULL, - PG_SJIS); + converted = UtfToLocal(src, len, dest, + &sjis_from_unicode_tree, + NULL, 0, + NULL, + PG_SJIS, + noError); - PG_RETURN_VOID(); + PG_RETURN_INT32(converted); } diff --git a/src/backend/utils/mb/conversion_procs/utf8_and_sjis2004/utf8_and_sjis2004.c b/src/backend/utils/mb/conversion_procs/utf8_and_sjis2004/utf8_and_sjis2004.c index 4bcc886d674..1fffdc5930c 100644 --- a/src/backend/utils/mb/conversion_procs/utf8_and_sjis2004/utf8_and_sjis2004.c +++ b/src/backend/utils/mb/conversion_procs/utf8_and_sjis2004/utf8_and_sjis2004.c @@ -28,8 +28,11 @@ PG_FUNCTION_INFO_V1(utf8_to_shift_jis_2004); * INTEGER, -- destination encoding id * CSTRING, -- source string (null terminated C string) * CSTRING, -- destination string (null terminated C string) - * INTEGER -- source string length - * ) returns VOID; + * INTEGER, -- source string length + * BOOL -- if true, don't throw an error if conversion fails + * ) returns INTEGER; + * + * Returns the number of bytes successfully converted. * ---------- */ Datum @@ -38,16 +41,19 @@ shift_jis_2004_to_utf8(PG_FUNCTION_ARGS) unsigned char *src = (unsigned char *) PG_GETARG_CSTRING(2); unsigned char *dest = (unsigned char *) PG_GETARG_CSTRING(3); int len = PG_GETARG_INT32(4); + bool noError = PG_GETARG_BOOL(5); + int converted; CHECK_ENCODING_CONVERSION_ARGS(PG_SHIFT_JIS_2004, PG_UTF8); - LocalToUtf(src, len, dest, - &shift_jis_2004_to_unicode_tree, - LUmapSHIFT_JIS_2004_combined, lengthof(LUmapSHIFT_JIS_2004_combined), - NULL, - PG_SHIFT_JIS_2004); + converted = LocalToUtf(src, len, dest, + &shift_jis_2004_to_unicode_tree, + LUmapSHIFT_JIS_2004_combined, lengthof(LUmapSHIFT_JIS_2004_combined), + NULL, + PG_SHIFT_JIS_2004, + noError); - PG_RETURN_VOID(); + PG_RETURN_INT32(converted); } Datum @@ -56,14 +62,17 @@ utf8_to_shift_jis_2004(PG_FUNCTION_ARGS) unsigned char *src = (unsigned char *) PG_GETARG_CSTRING(2); unsigned char *dest = (unsigned char *) PG_GETARG_CSTRING(3); int len = PG_GETARG_INT32(4); + bool noError = PG_GETARG_BOOL(5); + int converted; CHECK_ENCODING_CONVERSION_ARGS(PG_UTF8, PG_SHIFT_JIS_2004); - UtfToLocal(src, len, dest, - &shift_jis_2004_from_unicode_tree, - ULmapSHIFT_JIS_2004_combined, lengthof(ULmapSHIFT_JIS_2004_combined), - NULL, - PG_SHIFT_JIS_2004); + converted = UtfToLocal(src, len, dest, + &shift_jis_2004_from_unicode_tree, + ULmapSHIFT_JIS_2004_combined, lengthof(ULmapSHIFT_JIS_2004_combined), + NULL, + PG_SHIFT_JIS_2004, + noError); - PG_RETURN_VOID(); + PG_RETURN_INT32(converted); } diff --git a/src/backend/utils/mb/conversion_procs/utf8_and_uhc/utf8_and_uhc.c b/src/backend/utils/mb/conversion_procs/utf8_and_uhc/utf8_and_uhc.c index c8e512994a1..d9471dad097 100644 --- a/src/backend/utils/mb/conversion_procs/utf8_and_uhc/utf8_and_uhc.c +++ b/src/backend/utils/mb/conversion_procs/utf8_and_uhc/utf8_and_uhc.c @@ -28,8 +28,11 @@ PG_FUNCTION_INFO_V1(utf8_to_uhc); * INTEGER, -- destination encoding id * CSTRING, -- source string (null terminated C string) * CSTRING, -- destination string (null terminated C string) - * INTEGER -- source string length - * ) returns VOID; + * INTEGER, -- source string length + * BOOL -- if true, don't throw an error if conversion fails + * ) returns INTEGER; + * + * Returns the number of bytes successfully converted. * ---------- */ Datum @@ -38,16 +41,19 @@ uhc_to_utf8(PG_FUNCTION_ARGS) unsigned char *src = (unsigned char *) PG_GETARG_CSTRING(2); unsigned char *dest = (unsigned char *) PG_GETARG_CSTRING(3); int len = PG_GETARG_INT32(4); + bool noError = PG_GETARG_BOOL(5); + int converted; CHECK_ENCODING_CONVERSION_ARGS(PG_UHC, PG_UTF8); - LocalToUtf(src, len, dest, - &uhc_to_unicode_tree, - NULL, 0, - NULL, - PG_UHC); + converted = LocalToUtf(src, len, dest, + &uhc_to_unicode_tree, + NULL, 0, + NULL, + PG_UHC, + noError); - PG_RETURN_VOID(); + PG_RETURN_INT32(converted); } Datum @@ -56,14 +62,17 @@ utf8_to_uhc(PG_FUNCTION_ARGS) unsigned char *src = (unsigned char *) PG_GETARG_CSTRING(2); unsigned char *dest = (unsigned char *) PG_GETARG_CSTRING(3); int len = PG_GETARG_INT32(4); + bool noError = PG_GETARG_BOOL(5); + int converted; CHECK_ENCODING_CONVERSION_ARGS(PG_UTF8, PG_UHC); - UtfToLocal(src, len, dest, - &uhc_from_unicode_tree, - NULL, 0, - NULL, - PG_UHC); + converted = UtfToLocal(src, len, dest, + &uhc_from_unicode_tree, + NULL, 0, + NULL, + PG_UHC, + noError); - PG_RETURN_VOID(); + PG_RETURN_INT32(converted); } diff --git a/src/backend/utils/mb/conversion_procs/utf8_and_win/utf8_and_win.c b/src/backend/utils/mb/conversion_procs/utf8_and_win/utf8_and_win.c index 0c9493dee56..110ba5677d0 100644 --- a/src/backend/utils/mb/conversion_procs/utf8_and_win/utf8_and_win.c +++ b/src/backend/utils/mb/conversion_procs/utf8_and_win/utf8_and_win.c @@ -48,8 +48,11 @@ PG_FUNCTION_INFO_V1(utf8_to_win); * INTEGER, -- destination encoding id * CSTRING, -- source string (null terminated C string) * CSTRING, -- destination string (null terminated C string) - * INTEGER -- source string length - * ) returns VOID; + * INTEGER, -- source string length + * BOOL -- if true, don't throw an error if conversion fails + * ) returns INTEGER; + * + * Returns the number of bytes successfully converted. * ---------- */ @@ -81,6 +84,7 @@ win_to_utf8(PG_FUNCTION_ARGS) unsigned char *src = (unsigned char *) PG_GETARG_CSTRING(2); unsigned char *dest = (unsigned char *) PG_GETARG_CSTRING(3); int len = PG_GETARG_INT32(4); + bool noError = PG_GETARG_BOOL(5); int i; CHECK_ENCODING_CONVERSION_ARGS(-1, PG_UTF8); @@ -89,12 +93,15 @@ win_to_utf8(PG_FUNCTION_ARGS) { if (encoding == maps[i].encoding) { - LocalToUtf(src, len, dest, - maps[i].map1, - NULL, 0, - NULL, - encoding); - PG_RETURN_VOID(); + int converted; + + converted = LocalToUtf(src, len, dest, + maps[i].map1, + NULL, 0, + NULL, + encoding, + noError); + PG_RETURN_INT32(converted); } } @@ -103,7 +110,7 @@ win_to_utf8(PG_FUNCTION_ARGS) errmsg("unexpected encoding ID %d for WIN character sets", encoding))); - PG_RETURN_VOID(); + PG_RETURN_INT32(0); } Datum @@ -113,6 +120,7 @@ utf8_to_win(PG_FUNCTION_ARGS) unsigned char *src = (unsigned char *) PG_GETARG_CSTRING(2); unsigned char *dest = (unsigned char *) PG_GETARG_CSTRING(3); int len = PG_GETARG_INT32(4); + bool noError = PG_GETARG_BOOL(5); int i; CHECK_ENCODING_CONVERSION_ARGS(PG_UTF8, -1); @@ -121,12 +129,15 @@ utf8_to_win(PG_FUNCTION_ARGS) { if (encoding == maps[i].encoding) { - UtfToLocal(src, len, dest, - maps[i].map2, - NULL, 0, - NULL, - encoding); - PG_RETURN_VOID(); + int converted; + + converted = UtfToLocal(src, len, dest, + maps[i].map2, + NULL, 0, + NULL, + encoding, + noError); + PG_RETURN_INT32(converted); } } @@ -135,5 +146,5 @@ utf8_to_win(PG_FUNCTION_ARGS) errmsg("unexpected encoding ID %d for WIN character sets", encoding))); - PG_RETURN_VOID(); + PG_RETURN_INT32(0); } diff --git a/src/backend/utils/mb/mbutils.c b/src/backend/utils/mb/mbutils.c index 2578573b0ab..877d57eee53 100644 --- a/src/backend/utils/mb/mbutils.c +++ b/src/backend/utils/mb/mbutils.c @@ -406,12 +406,13 @@ pg_do_encoding_conversion(unsigned char *src, int len, MemoryContextAllocHuge(CurrentMemoryContext, (Size) len * MAX_CONVERSION_GROWTH + 1); - OidFunctionCall5(proc, - Int32GetDatum(src_encoding), - Int32GetDatum(dest_encoding), - CStringGetDatum(src), - CStringGetDatum(result), - Int32GetDatum(len)); + (void) OidFunctionCall6(proc, + Int32GetDatum(src_encoding), + Int32GetDatum(dest_encoding), + CStringGetDatum(src), + CStringGetDatum(result), + Int32GetDatum(len), + BoolGetDatum(false)); /* * If the result is large, it's worth repalloc'ing to release any extra @@ -435,6 +436,59 @@ pg_do_encoding_conversion(unsigned char *src, int len, return result; } +/* + * Convert src string to another encoding. + * + * This function has a different API than the other conversion functions. + * The caller should've looked up the conversion function using + * FindDefaultConversionProc(). Unlike the other functions, the converted + * result is not palloc'd. It is written to a caller-supplied buffer instead. + * + * src_encoding - encoding to convert from + * dest_encoding - encoding to convert to + * src, srclen - input buffer and its length in bytes + * dest, destlen - destination buffer and its size in bytes + * + * The output is null-terminated. + * + * If destlen < srclen * MAX_CONVERSION_LENGTH + 1, the converted output + * wouldn't necessarily fit in the output buffer, and the function will not + * convert the whole input. + * + * TODO: It would be nice to also return the number of bytes written to the + * caller, to avoid a call to strlen(). + */ +int +pg_do_encoding_conversion_buf(Oid proc, + int src_encoding, + int dest_encoding, + unsigned char *src, int srclen, + unsigned char *dest, int destlen, + bool noError) +{ + Datum result; + + /* + * If the destination buffer is not large enough to hold the result in the + * worst case, limit the input size passed to the conversion function. + * + * TODO: It would perhaps be more efficient to pass the destination buffer + * size to the conversion function, so that if the conversion expands less + * than the worst case, it could continue to fill up the whole buffer. + */ + if ((Size) srclen >= ((destlen - 1) / (Size) MAX_CONVERSION_GROWTH)) + srclen = ((destlen - 1) / (Size) MAX_CONVERSION_GROWTH); + + result = OidFunctionCall6(proc, + Int32GetDatum(src_encoding), + Int32GetDatum(dest_encoding), + CStringGetDatum(src), + CStringGetDatum(dest), + Int32GetDatum(srclen), + BoolGetDatum(noError)); + return DatumGetInt32(result); +} + /* * Convert string to encoding encoding_name. The source * encoding is the DB encoding. @@ -762,12 +816,13 @@ perform_default_encoding_conversion(const char *src, int len, MemoryContextAllocHuge(CurrentMemoryContext, (Size) len * MAX_CONVERSION_GROWTH + 1); - FunctionCall5(flinfo, + FunctionCall6(flinfo, Int32GetDatum(src_encoding), Int32GetDatum(dest_encoding), CStringGetDatum(src), CStringGetDatum(result), - Int32GetDatum(len)); + Int32GetDatum(len), + BoolGetDatum(false)); /* * Release extra space if there might be a lot --- see comments in @@ -849,12 +904,13 @@ pg_unicode_to_server(pg_wchar c, unsigned char *s) c_as_utf8[c_as_utf8_len] = '\0'; /* Convert, or throw error if we can't */ - FunctionCall5(Utf8ToServerConvProc, + FunctionCall6(Utf8ToServerConvProc, Int32GetDatum(PG_UTF8), Int32GetDatum(server_encoding), CStringGetDatum(c_as_utf8), CStringGetDatum(s), - Int32GetDatum(c_as_utf8_len)); + Int32GetDatum(c_as_utf8_len), + BoolGetDatum(false)); } diff --git a/src/bin/pg_upgrade/check.c b/src/bin/pg_upgrade/check.c index 43fc297eb69..ee6be95b08d 100644 --- a/src/bin/pg_upgrade/check.c +++ b/src/bin/pg_upgrade/check.c @@ -28,6 +28,7 @@ static void check_for_reg_data_type_usage(ClusterInfo *cluster); static void check_for_jsonb_9_4_usage(ClusterInfo *cluster); static void check_for_pg_role_prefix(ClusterInfo *cluster); static void check_for_new_tablespace_dir(ClusterInfo *new_cluster); +static void check_for_user_defined_encoding_conversions(ClusterInfo *cluster); static char *get_canonical_locale_name(int category, const char *locale); @@ -102,6 +103,15 @@ check_and_dump_old_cluster(bool live_check) check_for_reg_data_type_usage(&old_cluster); check_for_isn_and_int8_passing_mismatch(&old_cluster); + /* + * PG 14 changed the function signature of encoding conversion functions. + * Conversions from older versions cannot be upgraded automatically + * because the user-defined functions used by the encoding conversions + * need to changed to match the new signature. + */ + if (GET_MAJOR_VERSION(old_cluster.major_version) <= 1300) + check_for_user_defined_encoding_conversions(&old_cluster); + /* * Pre-PG 14 allowed user defined postfix operators, which are not * supported anymore. Verify there are none, iff applicable. @@ -1268,6 +1278,91 @@ check_for_pg_role_prefix(ClusterInfo *cluster) check_ok(); } +/* + * Verify that no user-defined encoding conversions exist. + */ +static void +check_for_user_defined_encoding_conversions(ClusterInfo *cluster) +{ + int dbnum; + FILE *script = NULL; + bool found = false; + char output_path[MAXPGPATH]; + + prep_status("Checking for user-defined encoding conversions"); + + snprintf(output_path, sizeof(output_path), + "encoding_conversions.txt"); + + /* Find any user defined encoding conversions */ + for (dbnum = 0; dbnum < cluster->dbarr.ndbs; dbnum++) + { + PGresult *res; + bool db_used = false; + int ntups; + int rowno; + int i_conoid, + i_conname, + i_nspname; + DbInfo *active_db = &cluster->dbarr.dbs[dbnum]; + PGconn *conn = connectToServer(cluster, active_db->db_name); + + /* + * The query below hardcodes FirstNormalObjectId as 16384 rather than + * interpolating that C #define into the query because, if that + * #define is ever changed, the cutoff we want to use is the value + * used by pre-version 14 servers, not that of some future version. + */ + res = executeQueryOrDie(conn, + "SELECT c.oid as conoid, c.conname, n.nspname " + "FROM pg_catalog.pg_conversion c, " + " pg_catalog.pg_namespace n " + "WHERE c.connamespace = n.oid AND " + " c.oid >= 16384"); + ntups = PQntuples(res); + i_conoid = PQfnumber(res, "conoid"); + i_conname = PQfnumber(res, "conname"); + i_nspname = PQfnumber(res, "nspname"); + for (rowno = 0; rowno < ntups; rowno++) + { + found = true; + if (script == NULL && + (script = fopen_priv(output_path, "w")) == NULL) + pg_fatal("could not open file \"%s\": %s\n", + output_path, strerror(errno)); + if (!db_used) + { + fprintf(script, "In database: %s\n", active_db->db_name); + db_used = true; + } + fprintf(script, " (oid=%s) %s.%s\n", + PQgetvalue(res, rowno, i_conoid), + PQgetvalue(res, rowno, i_nspname), + PQgetvalue(res, rowno, i_conname)); + } + + PQclear(res); + + PQfinish(conn); + } + + if (script) + fclose(script); + + if (found) + { + pg_log(PG_REPORT, "fatal\n"); + pg_fatal("Your installation contains user-defined encoding conversions.\n" + "The conversion function parameters changed in PostgreSQL version 14\n" + "so this cluster cannot currently be upgraded. You can remove the\n" + "encoding conversions in the old cluster and restart the upgrade.\n" + "A list of user-defined encoding conversions is in the file:\n" + " %s\n\n", output_path); + } + else + check_ok(); +} + /* * get_canonical_locale_name diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 4e0c9be58c3..12e228c7e06 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -10774,388 +10774,388 @@ # conversion functions { oid => '4302', descr => 'internal conversion function for KOI8R to MULE_INTERNAL', - proname => 'koi8r_to_mic', prolang => 'c', prorettype => 'void', - proargtypes => 'int4 int4 cstring internal int4', prosrc => 'koi8r_to_mic', + proname => 'koi8r_to_mic', prolang => 'c', prorettype => 'int4', + proargtypes => 'int4 int4 cstring internal int4 bool', prosrc => 'koi8r_to_mic', probin => '$libdir/cyrillic_and_mic' }, { oid => '4303', descr => 'internal conversion function for MULE_INTERNAL to KOI8R', - proname => 'mic_to_koi8r', prolang => 'c', prorettype => 'void', - proargtypes => 'int4 int4 cstring internal int4', prosrc => 'mic_to_koi8r', + proname => 'mic_to_koi8r', prolang => 'c', prorettype => 'int4', + proargtypes => 'int4 int4 cstring internal int4 bool', prosrc => 'mic_to_koi8r', probin => '$libdir/cyrillic_and_mic' }, { oid => '4304', descr => 'internal conversion function for ISO-8859-5 to MULE_INTERNAL', - proname => 'iso_to_mic', prolang => 'c', prorettype => 'void', - proargtypes => 'int4 int4 cstring internal int4', prosrc => 'iso_to_mic', + proname => 'iso_to_mic', prolang => 'c', prorettype => 'int4', + proargtypes => 'int4 int4 cstring internal int4 bool', prosrc => 'iso_to_mic', probin => '$libdir/cyrillic_and_mic' }, { oid => '4305', descr => 'internal conversion function for MULE_INTERNAL to ISO-8859-5', - proname => 'mic_to_iso', prolang => 'c', prorettype => 'void', - proargtypes => 'int4 int4 cstring internal int4', prosrc => 'mic_to_iso', + proname => 'mic_to_iso', prolang => 'c', prorettype => 'int4', + proargtypes => 'int4 int4 cstring internal int4 bool', prosrc => 'mic_to_iso', probin => '$libdir/cyrillic_and_mic' }, { oid => '4306', descr => 'internal conversion function for WIN1251 to MULE_INTERNAL', - proname => 'win1251_to_mic', prolang => 'c', prorettype => 'void', - proargtypes => 'int4 int4 cstring internal int4', prosrc => 'win1251_to_mic', + proname => 'win1251_to_mic', prolang => 'c', prorettype => 'int4', + proargtypes => 'int4 int4 cstring internal int4 bool', prosrc => 'win1251_to_mic', probin => '$libdir/cyrillic_and_mic' }, { oid => '4307', descr => 'internal conversion function for MULE_INTERNAL to WIN1251', - proname => 'mic_to_win1251', prolang => 'c', prorettype => 'void', - proargtypes => 'int4 int4 cstring internal int4', prosrc => 'mic_to_win1251', + proname => 'mic_to_win1251', prolang => 'c', prorettype => 'int4', + proargtypes => 'int4 int4 cstring internal int4 bool', prosrc => 'mic_to_win1251', probin => '$libdir/cyrillic_and_mic' }, { oid => '4308', descr => 'internal conversion function for WIN866 to MULE_INTERNAL', - proname => 'win866_to_mic', prolang => 'c', prorettype => 'void', - proargtypes => 'int4 int4 cstring internal int4', prosrc => 'win866_to_mic', + proname => 'win866_to_mic', prolang => 'c', prorettype => 'int4', + proargtypes => 'int4 int4 cstring internal int4 bool', prosrc => 'win866_to_mic', probin => '$libdir/cyrillic_and_mic' }, { oid => '4309', descr => 'internal conversion function for MULE_INTERNAL to WIN866', - proname => 'mic_to_win866', prolang => 'c', prorettype => 'void', - proargtypes => 'int4 int4 cstring internal int4', prosrc => 'mic_to_win866', + proname => 'mic_to_win866', prolang => 'c', prorettype => 'int4', + proargtypes => 'int4 int4 cstring internal int4 bool', prosrc => 'mic_to_win866', probin => '$libdir/cyrillic_and_mic' }, { oid => '4310', descr => 'internal conversion function for KOI8R to WIN1251', - proname => 'koi8r_to_win1251', prolang => 'c', prorettype => 'void', - proargtypes => 'int4 int4 cstring internal int4', + proname => 'koi8r_to_win1251', prolang => 'c', prorettype => 'int4', + proargtypes => 'int4 int4 cstring internal int4 bool', prosrc => 'koi8r_to_win1251', probin => '$libdir/cyrillic_and_mic' }, { oid => '4311', descr => 'internal conversion function for WIN1251 to KOI8R', - proname => 'win1251_to_koi8r', prolang => 'c', prorettype => 'void', - proargtypes => 'int4 int4 cstring internal int4', + proname => 'win1251_to_koi8r', prolang => 'c', prorettype => 'int4', + proargtypes => 'int4 int4 cstring internal int4 bool', prosrc => 'win1251_to_koi8r', probin => '$libdir/cyrillic_and_mic' }, { oid => '4312', descr => 'internal conversion function for KOI8R to WIN866', - proname => 'koi8r_to_win866', prolang => 'c', prorettype => 'void', - proargtypes => 'int4 int4 cstring internal int4', prosrc => 'koi8r_to_win866', + proname => 'koi8r_to_win866', prolang => 'c', prorettype => 'int4', + proargtypes => 'int4 int4 cstring internal int4 bool', prosrc => 'koi8r_to_win866', probin => '$libdir/cyrillic_and_mic' }, { oid => '4313', descr => 'internal conversion function for WIN866 to KOI8R', - proname => 'win866_to_koi8r', prolang => 'c', prorettype => 'void', - proargtypes => 'int4 int4 cstring internal int4', prosrc => 'win866_to_koi8r', + proname => 'win866_to_koi8r', prolang => 'c', prorettype => 'int4', + proargtypes => 'int4 int4 cstring internal int4 bool', prosrc => 'win866_to_koi8r', probin => '$libdir/cyrillic_and_mic' }, { oid => '4314', descr => 'internal conversion function for WIN866 to WIN1251', - proname => 'win866_to_win1251', prolang => 'c', prorettype => 'void', - proargtypes => 'int4 int4 cstring internal int4', + proname => 'win866_to_win1251', prolang => 'c', prorettype => 'int4', + proargtypes => 'int4 int4 cstring internal int4 bool', prosrc => 'win866_to_win1251', probin => '$libdir/cyrillic_and_mic' }, { oid => '4315', descr => 'internal conversion function for WIN1251 to WIN866', - proname => 'win1251_to_win866', prolang => 'c', prorettype => 'void', - proargtypes => 'int4 int4 cstring internal int4', + proname => 'win1251_to_win866', prolang => 'c', prorettype => 'int4', + proargtypes => 'int4 int4 cstring internal int4 bool', prosrc => 'win1251_to_win866', probin => '$libdir/cyrillic_and_mic' }, { oid => '4316', descr => 'internal conversion function for ISO-8859-5 to KOI8R', - proname => 'iso_to_koi8r', prolang => 'c', prorettype => 'void', - proargtypes => 'int4 int4 cstring internal int4', prosrc => 'iso_to_koi8r', + proname => 'iso_to_koi8r', prolang => 'c', prorettype => 'int4', + proargtypes => 'int4 int4 cstring internal int4 bool', prosrc => 'iso_to_koi8r', probin => '$libdir/cyrillic_and_mic' }, { oid => '4317', descr => 'internal conversion function for KOI8R to ISO-8859-5', - proname => 'koi8r_to_iso', prolang => 'c', prorettype => 'void', - proargtypes => 'int4 int4 cstring internal int4', prosrc => 'koi8r_to_iso', + proname => 'koi8r_to_iso', prolang => 'c', prorettype => 'int4', + proargtypes => 'int4 int4 cstring internal int4 bool', prosrc => 'koi8r_to_iso', probin => '$libdir/cyrillic_and_mic' }, { oid => '4318', descr => 'internal conversion function for ISO-8859-5 to WIN1251', - proname => 'iso_to_win1251', prolang => 'c', prorettype => 'void', - proargtypes => 'int4 int4 cstring internal int4', prosrc => 'iso_to_win1251', + proname => 'iso_to_win1251', prolang => 'c', prorettype => 'int4', + proargtypes => 'int4 int4 cstring internal int4 bool', prosrc => 'iso_to_win1251', probin => '$libdir/cyrillic_and_mic' }, { oid => '4319', descr => 'internal conversion function for WIN1251 to ISO-8859-5', - proname => 'win1251_to_iso', prolang => 'c', prorettype => 'void', - proargtypes => 'int4 int4 cstring internal int4', prosrc => 'win1251_to_iso', + proname => 'win1251_to_iso', prolang => 'c', prorettype => 'int4', + proargtypes => 'int4 int4 cstring internal int4 bool', prosrc => 'win1251_to_iso', probin => '$libdir/cyrillic_and_mic' }, { oid => '4320', descr => 'internal conversion function for ISO-8859-5 to WIN866', - proname => 'iso_to_win866', prolang => 'c', prorettype => 'void', - proargtypes => 'int4 int4 cstring internal int4', prosrc => 'iso_to_win866', + proname => 'iso_to_win866', prolang => 'c', prorettype => 'int4', + proargtypes => 'int4 int4 cstring internal int4 bool', prosrc => 'iso_to_win866', probin => '$libdir/cyrillic_and_mic' }, { oid => '4321', descr => 'internal conversion function for WIN866 to ISO-8859-5', - proname => 'win866_to_iso', prolang => 'c', prorettype => 'void', - proargtypes => 'int4 int4 cstring internal int4', prosrc => 'win866_to_iso', + proname => 'win866_to_iso', prolang => 'c', prorettype => 'int4', + proargtypes => 'int4 int4 cstring internal int4 bool', prosrc => 'win866_to_iso', probin => '$libdir/cyrillic_and_mic' }, { oid => '4322', descr => 'internal conversion function for EUC_CN to MULE_INTERNAL', - proname => 'euc_cn_to_mic', prolang => 'c', prorettype => 'void', - proargtypes => 'int4 int4 cstring internal int4', prosrc => 'euc_cn_to_mic', + proname => 'euc_cn_to_mic', prolang => 'c', prorettype => 'int4', + proargtypes => 'int4 int4 cstring internal int4 bool', prosrc => 'euc_cn_to_mic', probin => '$libdir/euc_cn_and_mic' }, { oid => '4323', descr => 'internal conversion function for MULE_INTERNAL to EUC_CN', - proname => 'mic_to_euc_cn', prolang => 'c', prorettype => 'void', - proargtypes => 'int4 int4 cstring internal int4', prosrc => 'mic_to_euc_cn', + proname => 'mic_to_euc_cn', prolang => 'c', prorettype => 'int4', + proargtypes => 'int4 int4 cstring internal int4 bool', prosrc => 'mic_to_euc_cn', probin => '$libdir/euc_cn_and_mic' }, { oid => '4324', descr => 'internal conversion function for EUC_JP to SJIS', - proname => 'euc_jp_to_sjis', prolang => 'c', prorettype => 'void', - proargtypes => 'int4 int4 cstring internal int4', prosrc => 'euc_jp_to_sjis', + proname => 'euc_jp_to_sjis', prolang => 'c', prorettype => 'int4', + proargtypes => 'int4 int4 cstring internal int4 bool', prosrc => 'euc_jp_to_sjis', probin => '$libdir/euc_jp_and_sjis' }, { oid => '4325', descr => 'internal conversion function for SJIS to EUC_JP', - proname => 'sjis_to_euc_jp', prolang => 'c', prorettype => 'void', - proargtypes => 'int4 int4 cstring internal int4', prosrc => 'sjis_to_euc_jp', + proname => 'sjis_to_euc_jp', prolang => 'c', prorettype => 'int4', + proargtypes => 'int4 int4 cstring internal int4 bool', prosrc => 'sjis_to_euc_jp', probin => '$libdir/euc_jp_and_sjis' }, { oid => '4326', descr => 'internal conversion function for EUC_JP to MULE_INTERNAL', - proname => 'euc_jp_to_mic', prolang => 'c', prorettype => 'void', - proargtypes => 'int4 int4 cstring internal int4', prosrc => 'euc_jp_to_mic', + proname => 'euc_jp_to_mic', prolang => 'c', prorettype => 'int4', + proargtypes => 'int4 int4 cstring internal int4 bool', prosrc => 'euc_jp_to_mic', probin => '$libdir/euc_jp_and_sjis' }, { oid => '4327', descr => 'internal conversion function for SJIS to MULE_INTERNAL', - proname => 'sjis_to_mic', prolang => 'c', prorettype => 'void', - proargtypes => 'int4 int4 cstring internal int4', prosrc => 'sjis_to_mic', + proname => 'sjis_to_mic', prolang => 'c', prorettype => 'int4', + proargtypes => 'int4 int4 cstring internal int4 bool', prosrc => 'sjis_to_mic', probin => '$libdir/euc_jp_and_sjis' }, { oid => '4328', descr => 'internal conversion function for MULE_INTERNAL to EUC_JP', - proname => 'mic_to_euc_jp', prolang => 'c', prorettype => 'void', - proargtypes => 'int4 int4 cstring internal int4', prosrc => 'mic_to_euc_jp', + proname => 'mic_to_euc_jp', prolang => 'c', prorettype => 'int4', + proargtypes => 'int4 int4 cstring internal int4 bool', prosrc => 'mic_to_euc_jp', probin => '$libdir/euc_jp_and_sjis' }, { oid => '4329', descr => 'internal conversion function for MULE_INTERNAL to SJIS', - proname => 'mic_to_sjis', prolang => 'c', prorettype => 'void', - proargtypes => 'int4 int4 cstring internal int4', prosrc => 'mic_to_sjis', + proname => 'mic_to_sjis', prolang => 'c', prorettype => 'int4', + proargtypes => 'int4 int4 cstring internal int4 bool', prosrc => 'mic_to_sjis', probin => '$libdir/euc_jp_and_sjis' }, { oid => '4330', descr => 'internal conversion function for EUC_KR to MULE_INTERNAL', - proname => 'euc_kr_to_mic', prolang => 'c', prorettype => 'void', - proargtypes => 'int4 int4 cstring internal int4', prosrc => 'euc_kr_to_mic', + proname => 'euc_kr_to_mic', prolang => 'c', prorettype => 'int4', + proargtypes => 'int4 int4 cstring internal int4 bool', prosrc => 'euc_kr_to_mic', probin => '$libdir/euc_kr_and_mic' }, { oid => '4331', descr => 'internal conversion function for MULE_INTERNAL to EUC_KR', - proname => 'mic_to_euc_kr', prolang => 'c', prorettype => 'void', - proargtypes => 'int4 int4 cstring internal int4', prosrc => 'mic_to_euc_kr', + proname => 'mic_to_euc_kr', prolang => 'c', prorettype => 'int4', + proargtypes => 'int4 int4 cstring internal int4 bool', prosrc => 'mic_to_euc_kr', probin => '$libdir/euc_kr_and_mic' }, { oid => '4332', descr => 'internal conversion function for EUC_TW to BIG5', - proname => 'euc_tw_to_big5', prolang => 'c', prorettype => 'void', - proargtypes => 'int4 int4 cstring internal int4', prosrc => 'euc_tw_to_big5', + proname => 'euc_tw_to_big5', prolang => 'c', prorettype => 'int4', + proargtypes => 'int4 int4 cstring internal int4 bool', prosrc => 'euc_tw_to_big5', probin => '$libdir/euc_tw_and_big5' }, { oid => '4333', descr => 'internal conversion function for BIG5 to EUC_TW', - proname => 'big5_to_euc_tw', prolang => 'c', prorettype => 'void', - proargtypes => 'int4 int4 cstring internal int4', prosrc => 'big5_to_euc_tw', + proname => 'big5_to_euc_tw', prolang => 'c', prorettype => 'int4', + proargtypes => 'int4 int4 cstring internal int4 bool', prosrc => 'big5_to_euc_tw', probin => '$libdir/euc_tw_and_big5' }, { oid => '4334', descr => 'internal conversion function for EUC_TW to MULE_INTERNAL', - proname => 'euc_tw_to_mic', prolang => 'c', prorettype => 'void', - proargtypes => 'int4 int4 cstring internal int4', prosrc => 'euc_tw_to_mic', + proname => 'euc_tw_to_mic', prolang => 'c', prorettype => 'int4', + proargtypes => 'int4 int4 cstring internal int4 bool', prosrc => 'euc_tw_to_mic', probin => '$libdir/euc_tw_and_big5' }, { oid => '4335', descr => 'internal conversion function for BIG5 to MULE_INTERNAL', - proname => 'big5_to_mic', prolang => 'c', prorettype => 'void', - proargtypes => 'int4 int4 cstring internal int4', prosrc => 'big5_to_mic', + proname => 'big5_to_mic', prolang => 'c', prorettype => 'int4', + proargtypes => 'int4 int4 cstring internal int4 bool', prosrc => 'big5_to_mic', probin => '$libdir/euc_tw_and_big5' }, { oid => '4336', descr => 'internal conversion function for MULE_INTERNAL to EUC_TW', - proname => 'mic_to_euc_tw', prolang => 'c', prorettype => 'void', - proargtypes => 'int4 int4 cstring internal int4', prosrc => 'mic_to_euc_tw', + proname => 'mic_to_euc_tw', prolang => 'c', prorettype => 'int4', + proargtypes => 'int4 int4 cstring internal int4 bool', prosrc => 'mic_to_euc_tw', probin => '$libdir/euc_tw_and_big5' }, { oid => '4337', descr => 'internal conversion function for MULE_INTERNAL to BIG5', - proname => 'mic_to_big5', prolang => 'c', prorettype => 'void', - proargtypes => 'int4 int4 cstring internal int4', prosrc => 'mic_to_big5', + proname => 'mic_to_big5', prolang => 'c', prorettype => 'int4', + proargtypes => 'int4 int4 cstring internal int4 bool', prosrc => 'mic_to_big5', probin => '$libdir/euc_tw_and_big5' }, { oid => '4338', descr => 'internal conversion function for LATIN2 to MULE_INTERNAL', - proname => 'latin2_to_mic', prolang => 'c', prorettype => 'void', - proargtypes => 'int4 int4 cstring internal int4', prosrc => 'latin2_to_mic', + proname => 'latin2_to_mic', prolang => 'c', prorettype => 'int4', + proargtypes => 'int4 int4 cstring internal int4 bool', prosrc => 'latin2_to_mic', probin => '$libdir/latin2_and_win1250' }, { oid => '4339', descr => 'internal conversion function for MULE_INTERNAL to LATIN2', - proname => 'mic_to_latin2', prolang => 'c', prorettype => 'void', - proargtypes => 'int4 int4 cstring internal int4', prosrc => 'mic_to_latin2', + proname => 'mic_to_latin2', prolang => 'c', prorettype => 'int4', + proargtypes => 'int4 int4 cstring internal int4 bool', prosrc => 'mic_to_latin2', probin => '$libdir/latin2_and_win1250' }, { oid => '4340', descr => 'internal conversion function for WIN1250 to MULE_INTERNAL', - proname => 'win1250_to_mic', prolang => 'c', prorettype => 'void', - proargtypes => 'int4 int4 cstring internal int4', prosrc => 'win1250_to_mic', + proname => 'win1250_to_mic', prolang => 'c', prorettype => 'int4', + proargtypes => 'int4 int4 cstring internal int4 bool', prosrc => 'win1250_to_mic', probin => '$libdir/latin2_and_win1250' }, { oid => '4341', descr => 'internal conversion function for MULE_INTERNAL to WIN1250', - proname => 'mic_to_win1250', prolang => 'c', prorettype => 'void', - proargtypes => 'int4 int4 cstring internal int4', prosrc => 'mic_to_win1250', + proname => 'mic_to_win1250', prolang => 'c', prorettype => 'int4', + proargtypes => 'int4 int4 cstring internal int4 bool', prosrc => 'mic_to_win1250', probin => '$libdir/latin2_and_win1250' }, { oid => '4342', descr => 'internal conversion function for LATIN2 to WIN1250', - proname => 'latin2_to_win1250', prolang => 'c', prorettype => 'void', - proargtypes => 'int4 int4 cstring internal int4', + proname => 'latin2_to_win1250', prolang => 'c', prorettype => 'int4', + proargtypes => 'int4 int4 cstring internal int4 bool', prosrc => 'latin2_to_win1250', probin => '$libdir/latin2_and_win1250' }, { oid => '4343', descr => 'internal conversion function for WIN1250 to LATIN2', - proname => 'win1250_to_latin2', prolang => 'c', prorettype => 'void', - proargtypes => 'int4 int4 cstring internal int4', + proname => 'win1250_to_latin2', prolang => 'c', prorettype => 'int4', + proargtypes => 'int4 int4 cstring internal int4 bool', prosrc => 'win1250_to_latin2', probin => '$libdir/latin2_and_win1250' }, { oid => '4344', descr => 'internal conversion function for LATIN1 to MULE_INTERNAL', - proname => 'latin1_to_mic', prolang => 'c', prorettype => 'void', - proargtypes => 'int4 int4 cstring internal int4', prosrc => 'latin1_to_mic', + proname => 'latin1_to_mic', prolang => 'c', prorettype => 'int4', + proargtypes => 'int4 int4 cstring internal int4 bool', prosrc => 'latin1_to_mic', probin => '$libdir/latin_and_mic' }, { oid => '4345', descr => 'internal conversion function for MULE_INTERNAL to LATIN1', - proname => 'mic_to_latin1', prolang => 'c', prorettype => 'void', - proargtypes => 'int4 int4 cstring internal int4', prosrc => 'mic_to_latin1', + proname => 'mic_to_latin1', prolang => 'c', prorettype => 'int4', + proargtypes => 'int4 int4 cstring internal int4 bool', prosrc => 'mic_to_latin1', probin => '$libdir/latin_and_mic' }, { oid => '4346', descr => 'internal conversion function for LATIN3 to MULE_INTERNAL', - proname => 'latin3_to_mic', prolang => 'c', prorettype => 'void', - proargtypes => 'int4 int4 cstring internal int4', prosrc => 'latin3_to_mic', + proname => 'latin3_to_mic', prolang => 'c', prorettype => 'int4', + proargtypes => 'int4 int4 cstring internal int4 bool', prosrc => 'latin3_to_mic', probin => '$libdir/latin_and_mic' }, { oid => '4347', descr => 'internal conversion function for MULE_INTERNAL to LATIN3', - proname => 'mic_to_latin3', prolang => 'c', prorettype => 'void', - proargtypes => 'int4 int4 cstring internal int4', prosrc => 'mic_to_latin3', + proname => 'mic_to_latin3', prolang => 'c', prorettype => 'int4', + proargtypes => 'int4 int4 cstring internal int4 bool', prosrc => 'mic_to_latin3', probin => '$libdir/latin_and_mic' }, { oid => '4348', descr => 'internal conversion function for LATIN4 to MULE_INTERNAL', - proname => 'latin4_to_mic', prolang => 'c', prorettype => 'void', - proargtypes => 'int4 int4 cstring internal int4', prosrc => 'latin4_to_mic', + proname => 'latin4_to_mic', prolang => 'c', prorettype => 'int4', + proargtypes => 'int4 int4 cstring internal int4 bool', prosrc => 'latin4_to_mic', probin => '$libdir/latin_and_mic' }, { oid => '4349', descr => 'internal conversion function for MULE_INTERNAL to LATIN4', - proname => 'mic_to_latin4', prolang => 'c', prorettype => 'void', - proargtypes => 'int4 int4 cstring internal int4', prosrc => 'mic_to_latin4', + proname => 'mic_to_latin4', prolang => 'c', prorettype => 'int4', + proargtypes => 'int4 int4 cstring internal int4 bool', prosrc => 'mic_to_latin4', probin => '$libdir/latin_and_mic' }, { oid => '4352', descr => 'internal conversion function for BIG5 to UTF8', - proname => 'big5_to_utf8', prolang => 'c', prorettype => 'void', - proargtypes => 'int4 int4 cstring internal int4', prosrc => 'big5_to_utf8', + proname => 'big5_to_utf8', prolang => 'c', prorettype => 'int4', + proargtypes => 'int4 int4 cstring internal int4 bool', prosrc => 'big5_to_utf8', probin => '$libdir/utf8_and_big5' }, { oid => '4353', descr => 'internal conversion function for UTF8 to BIG5', - proname => 'utf8_to_big5', prolang => 'c', prorettype => 'void', - proargtypes => 'int4 int4 cstring internal int4', prosrc => 'utf8_to_big5', + proname => 'utf8_to_big5', prolang => 'c', prorettype => 'int4', + proargtypes => 'int4 int4 cstring internal int4 bool', prosrc => 'utf8_to_big5', probin => '$libdir/utf8_and_big5' }, { oid => '4354', descr => 'internal conversion function for UTF8 to KOI8R', - proname => 'utf8_to_koi8r', prolang => 'c', prorettype => 'void', - proargtypes => 'int4 int4 cstring internal int4', prosrc => 'utf8_to_koi8r', + proname => 'utf8_to_koi8r', prolang => 'c', prorettype => 'int4', + proargtypes => 'int4 int4 cstring internal int4 bool', prosrc => 'utf8_to_koi8r', probin => '$libdir/utf8_and_cyrillic' }, { oid => '4355', descr => 'internal conversion function for KOI8R to UTF8', - proname => 'koi8r_to_utf8', prolang => 'c', prorettype => 'void', - proargtypes => 'int4 int4 cstring internal int4', prosrc => 'koi8r_to_utf8', + proname => 'koi8r_to_utf8', prolang => 'c', prorettype => 'int4', + proargtypes => 'int4 int4 cstring internal int4 bool', prosrc => 'koi8r_to_utf8', probin => '$libdir/utf8_and_cyrillic' }, { oid => '4356', descr => 'internal conversion function for UTF8 to KOI8U', - proname => 'utf8_to_koi8u', prolang => 'c', prorettype => 'void', - proargtypes => 'int4 int4 cstring internal int4', prosrc => 'utf8_to_koi8u', + proname => 'utf8_to_koi8u', prolang => 'c', prorettype => 'int4', + proargtypes => 'int4 int4 cstring internal int4 bool', prosrc => 'utf8_to_koi8u', probin => '$libdir/utf8_and_cyrillic' }, { oid => '4357', descr => 'internal conversion function for KOI8U to UTF8', - proname => 'koi8u_to_utf8', prolang => 'c', prorettype => 'void', - proargtypes => 'int4 int4 cstring internal int4', prosrc => 'koi8u_to_utf8', + proname => 'koi8u_to_utf8', prolang => 'c', prorettype => 'int4', + proargtypes => 'int4 int4 cstring internal int4 bool', prosrc => 'koi8u_to_utf8', probin => '$libdir/utf8_and_cyrillic' }, { oid => '4358', descr => 'internal conversion function for UTF8 to WIN', - proname => 'utf8_to_win', prolang => 'c', prorettype => 'void', - proargtypes => 'int4 int4 cstring internal int4', prosrc => 'utf8_to_win', + proname => 'utf8_to_win', prolang => 'c', prorettype => 'int4', + proargtypes => 'int4 int4 cstring internal int4 bool', prosrc => 'utf8_to_win', probin => '$libdir/utf8_and_win' }, { oid => '4359', descr => 'internal conversion function for WIN to UTF8', - proname => 'win_to_utf8', prolang => 'c', prorettype => 'void', - proargtypes => 'int4 int4 cstring internal int4', prosrc => 'win_to_utf8', + proname => 'win_to_utf8', prolang => 'c', prorettype => 'int4', + proargtypes => 'int4 int4 cstring internal int4 bool', prosrc => 'win_to_utf8', probin => '$libdir/utf8_and_win' }, { oid => '4360', descr => 'internal conversion function for EUC_CN to UTF8', - proname => 'euc_cn_to_utf8', prolang => 'c', prorettype => 'void', - proargtypes => 'int4 int4 cstring internal int4', prosrc => 'euc_cn_to_utf8', + proname => 'euc_cn_to_utf8', prolang => 'c', prorettype => 'int4', + proargtypes => 'int4 int4 cstring internal int4 bool', prosrc => 'euc_cn_to_utf8', probin => '$libdir/utf8_and_euc_cn' }, { oid => '4361', descr => 'internal conversion function for UTF8 to EUC_CN', - proname => 'utf8_to_euc_cn', prolang => 'c', prorettype => 'void', - proargtypes => 'int4 int4 cstring internal int4', prosrc => 'utf8_to_euc_cn', + proname => 'utf8_to_euc_cn', prolang => 'c', prorettype => 'int4', + proargtypes => 'int4 int4 cstring internal int4 bool', prosrc => 'utf8_to_euc_cn', probin => '$libdir/utf8_and_euc_cn' }, { oid => '4362', descr => 'internal conversion function for EUC_JP to UTF8', - proname => 'euc_jp_to_utf8', prolang => 'c', prorettype => 'void', - proargtypes => 'int4 int4 cstring internal int4', prosrc => 'euc_jp_to_utf8', + proname => 'euc_jp_to_utf8', prolang => 'c', prorettype => 'int4', + proargtypes => 'int4 int4 cstring internal int4 bool', prosrc => 'euc_jp_to_utf8', probin => '$libdir/utf8_and_euc_jp' }, { oid => '4363', descr => 'internal conversion function for UTF8 to EUC_JP', - proname => 'utf8_to_euc_jp', prolang => 'c', prorettype => 'void', - proargtypes => 'int4 int4 cstring internal int4', prosrc => 'utf8_to_euc_jp', + proname => 'utf8_to_euc_jp', prolang => 'c', prorettype => 'int4', + proargtypes => 'int4 int4 cstring internal int4 bool', prosrc => 'utf8_to_euc_jp', probin => '$libdir/utf8_and_euc_jp' }, { oid => '4364', descr => 'internal conversion function for EUC_KR to UTF8', - proname => 'euc_kr_to_utf8', prolang => 'c', prorettype => 'void', - proargtypes => 'int4 int4 cstring internal int4', prosrc => 'euc_kr_to_utf8', + proname => 'euc_kr_to_utf8', prolang => 'c', prorettype => 'int4', + proargtypes => 'int4 int4 cstring internal int4 bool', prosrc => 'euc_kr_to_utf8', probin => '$libdir/utf8_and_euc_kr' }, { oid => '4365', descr => 'internal conversion function for UTF8 to EUC_KR', - proname => 'utf8_to_euc_kr', prolang => 'c', prorettype => 'void', - proargtypes => 'int4 int4 cstring internal int4', prosrc => 'utf8_to_euc_kr', + proname => 'utf8_to_euc_kr', prolang => 'c', prorettype => 'int4', + proargtypes => 'int4 int4 cstring internal int4 bool', prosrc => 'utf8_to_euc_kr', probin => '$libdir/utf8_and_euc_kr' }, { oid => '4366', descr => 'internal conversion function for EUC_TW to UTF8', - proname => 'euc_tw_to_utf8', prolang => 'c', prorettype => 'void', - proargtypes => 'int4 int4 cstring internal int4', prosrc => 'euc_tw_to_utf8', + proname => 'euc_tw_to_utf8', prolang => 'c', prorettype => 'int4', + proargtypes => 'int4 int4 cstring internal int4 bool', prosrc => 'euc_tw_to_utf8', probin => '$libdir/utf8_and_euc_tw' }, { oid => '4367', descr => 'internal conversion function for UTF8 to EUC_TW', - proname => 'utf8_to_euc_tw', prolang => 'c', prorettype => 'void', - proargtypes => 'int4 int4 cstring internal int4', prosrc => 'utf8_to_euc_tw', + proname => 'utf8_to_euc_tw', prolang => 'c', prorettype => 'int4', + proargtypes => 'int4 int4 cstring internal int4 bool', prosrc => 'utf8_to_euc_tw', probin => '$libdir/utf8_and_euc_tw' }, { oid => '4368', descr => 'internal conversion function for GB18030 to UTF8', - proname => 'gb18030_to_utf8', prolang => 'c', prorettype => 'void', - proargtypes => 'int4 int4 cstring internal int4', prosrc => 'gb18030_to_utf8', + proname => 'gb18030_to_utf8', prolang => 'c', prorettype => 'int4', + proargtypes => 'int4 int4 cstring internal int4 bool', prosrc => 'gb18030_to_utf8', probin => '$libdir/utf8_and_gb18030' }, { oid => '4369', descr => 'internal conversion function for UTF8 to GB18030', - proname => 'utf8_to_gb18030', prolang => 'c', prorettype => 'void', - proargtypes => 'int4 int4 cstring internal int4', prosrc => 'utf8_to_gb18030', + proname => 'utf8_to_gb18030', prolang => 'c', prorettype => 'int4', + proargtypes => 'int4 int4 cstring internal int4 bool', prosrc => 'utf8_to_gb18030', probin => '$libdir/utf8_and_gb18030' }, { oid => '4370', descr => 'internal conversion function for GBK to UTF8', - proname => 'gbk_to_utf8', prolang => 'c', prorettype => 'void', - proargtypes => 'int4 int4 cstring internal int4', prosrc => 'gbk_to_utf8', + proname => 'gbk_to_utf8', prolang => 'c', prorettype => 'int4', + proargtypes => 'int4 int4 cstring internal int4 bool', prosrc => 'gbk_to_utf8', probin => '$libdir/utf8_and_gbk' }, { oid => '4371', descr => 'internal conversion function for UTF8 to GBK', - proname => 'utf8_to_gbk', prolang => 'c', prorettype => 'void', - proargtypes => 'int4 int4 cstring internal int4', prosrc => 'utf8_to_gbk', + proname => 'utf8_to_gbk', prolang => 'c', prorettype => 'int4', + proargtypes => 'int4 int4 cstring internal int4 bool', prosrc => 'utf8_to_gbk', probin => '$libdir/utf8_and_gbk' }, { oid => '4372', descr => 'internal conversion function for UTF8 to ISO-8859 2-16', - proname => 'utf8_to_iso8859', prolang => 'c', prorettype => 'void', - proargtypes => 'int4 int4 cstring internal int4', prosrc => 'utf8_to_iso8859', + proname => 'utf8_to_iso8859', prolang => 'c', prorettype => 'int4', + proargtypes => 'int4 int4 cstring internal int4 bool', prosrc => 'utf8_to_iso8859', probin => '$libdir/utf8_and_iso8859' }, { oid => '4373', descr => 'internal conversion function for ISO-8859 2-16 to UTF8', - proname => 'iso8859_to_utf8', prolang => 'c', prorettype => 'void', - proargtypes => 'int4 int4 cstring internal int4', prosrc => 'iso8859_to_utf8', + proname => 'iso8859_to_utf8', prolang => 'c', prorettype => 'int4', + proargtypes => 'int4 int4 cstring internal int4 bool', prosrc => 'iso8859_to_utf8', probin => '$libdir/utf8_and_iso8859' }, { oid => '4374', descr => 'internal conversion function for LATIN1 to UTF8', - proname => 'iso8859_1_to_utf8', prolang => 'c', prorettype => 'void', - proargtypes => 'int4 int4 cstring internal int4', + proname => 'iso8859_1_to_utf8', prolang => 'c', prorettype => 'int4', + proargtypes => 'int4 int4 cstring internal int4 bool', prosrc => 'iso8859_1_to_utf8', probin => '$libdir/utf8_and_iso8859_1' }, { oid => '4375', descr => 'internal conversion function for UTF8 to LATIN1', - proname => 'utf8_to_iso8859_1', prolang => 'c', prorettype => 'void', - proargtypes => 'int4 int4 cstring internal int4', + proname => 'utf8_to_iso8859_1', prolang => 'c', prorettype => 'int4', + proargtypes => 'int4 int4 cstring internal int4 bool', prosrc => 'utf8_to_iso8859_1', probin => '$libdir/utf8_and_iso8859_1' }, { oid => '4376', descr => 'internal conversion function for JOHAB to UTF8', - proname => 'johab_to_utf8', prolang => 'c', prorettype => 'void', - proargtypes => 'int4 int4 cstring internal int4', prosrc => 'johab_to_utf8', + proname => 'johab_to_utf8', prolang => 'c', prorettype => 'int4', + proargtypes => 'int4 int4 cstring internal int4 bool', prosrc => 'johab_to_utf8', probin => '$libdir/utf8_and_johab' }, { oid => '4377', descr => 'internal conversion function for UTF8 to JOHAB', - proname => 'utf8_to_johab', prolang => 'c', prorettype => 'void', - proargtypes => 'int4 int4 cstring internal int4', prosrc => 'utf8_to_johab', + proname => 'utf8_to_johab', prolang => 'c', prorettype => 'int4', + proargtypes => 'int4 int4 cstring internal int4 bool', prosrc => 'utf8_to_johab', probin => '$libdir/utf8_and_johab' }, { oid => '4378', descr => 'internal conversion function for SJIS to UTF8', - proname => 'sjis_to_utf8', prolang => 'c', prorettype => 'void', - proargtypes => 'int4 int4 cstring internal int4', prosrc => 'sjis_to_utf8', + proname => 'sjis_to_utf8', prolang => 'c', prorettype => 'int4', + proargtypes => 'int4 int4 cstring internal int4 bool', prosrc => 'sjis_to_utf8', probin => '$libdir/utf8_and_sjis' }, { oid => '4379', descr => 'internal conversion function for UTF8 to SJIS', - proname => 'utf8_to_sjis', prolang => 'c', prorettype => 'void', - proargtypes => 'int4 int4 cstring internal int4', prosrc => 'utf8_to_sjis', + proname => 'utf8_to_sjis', prolang => 'c', prorettype => 'int4', + proargtypes => 'int4 int4 cstring internal int4 bool', prosrc => 'utf8_to_sjis', probin => '$libdir/utf8_and_sjis' }, { oid => '4380', descr => 'internal conversion function for UHC to UTF8', - proname => 'uhc_to_utf8', prolang => 'c', prorettype => 'void', - proargtypes => 'int4 int4 cstring internal int4', prosrc => 'uhc_to_utf8', + proname => 'uhc_to_utf8', prolang => 'c', prorettype => 'int4', + proargtypes => 'int4 int4 cstring internal int4 bool', prosrc => 'uhc_to_utf8', probin => '$libdir/utf8_and_uhc' }, { oid => '4381', descr => 'internal conversion function for UTF8 to UHC', - proname => 'utf8_to_uhc', prolang => 'c', prorettype => 'void', - proargtypes => 'int4 int4 cstring internal int4', prosrc => 'utf8_to_uhc', + proname => 'utf8_to_uhc', prolang => 'c', prorettype => 'int4', + proargtypes => 'int4 int4 cstring internal int4 bool', prosrc => 'utf8_to_uhc', probin => '$libdir/utf8_and_uhc' }, { oid => '4382', descr => 'internal conversion function for EUC_JIS_2004 to UTF8', - proname => 'euc_jis_2004_to_utf8', prolang => 'c', prorettype => 'void', - proargtypes => 'int4 int4 cstring internal int4', + proname => 'euc_jis_2004_to_utf8', prolang => 'c', prorettype => 'int4', + proargtypes => 'int4 int4 cstring internal int4 bool', prosrc => 'euc_jis_2004_to_utf8', probin => '$libdir/utf8_and_euc2004' }, { oid => '4383', descr => 'internal conversion function for UTF8 to EUC_JIS_2004', - proname => 'utf8_to_euc_jis_2004', prolang => 'c', prorettype => 'void', - proargtypes => 'int4 int4 cstring internal int4', + proname => 'utf8_to_euc_jis_2004', prolang => 'c', prorettype => 'int4', + proargtypes => 'int4 int4 cstring internal int4 bool', prosrc => 'utf8_to_euc_jis_2004', probin => '$libdir/utf8_and_euc2004' }, { oid => '4384', descr => 'internal conversion function for SHIFT_JIS_2004 to UTF8', - proname => 'shift_jis_2004_to_utf8', prolang => 'c', prorettype => 'void', - proargtypes => 'int4 int4 cstring internal int4', + proname => 'shift_jis_2004_to_utf8', prolang => 'c', prorettype => 'int4', + proargtypes => 'int4 int4 cstring internal int4 bool', prosrc => 'shift_jis_2004_to_utf8', probin => '$libdir/utf8_and_sjis2004' }, { oid => '4385', descr => 'internal conversion function for UTF8 to SHIFT_JIS_2004', - proname => 'utf8_to_shift_jis_2004', prolang => 'c', prorettype => 'void', - proargtypes => 'int4 int4 cstring internal int4', + proname => 'utf8_to_shift_jis_2004', prolang => 'c', prorettype => 'int4', + proargtypes => 'int4 int4 cstring internal int4 bool', prosrc => 'utf8_to_shift_jis_2004', probin => '$libdir/utf8_and_sjis2004' }, { oid => '4386', descr => 'internal conversion function for EUC_JIS_2004 to SHIFT_JIS_2004', proname => 'euc_jis_2004_to_shift_jis_2004', prolang => 'c', - prorettype => 'void', proargtypes => 'int4 int4 cstring internal int4', + prorettype => 'int4', proargtypes => 'int4 int4 cstring internal int4 bool', prosrc => 'euc_jis_2004_to_shift_jis_2004', probin => '$libdir/euc2004_sjis2004' }, { oid => '4387', descr => 'internal conversion function for SHIFT_JIS_2004 to EUC_JIS_2004', proname => 'shift_jis_2004_to_euc_jis_2004', prolang => 'c', - prorettype => 'void', proargtypes => 'int4 int4 cstring internal int4', + prorettype => 'int4', proargtypes => 'int4 int4 cstring internal int4 bool', prosrc => 'shift_jis_2004_to_euc_jis_2004', probin => '$libdir/euc2004_sjis2004' }, diff --git a/src/include/mb/pg_wchar.h b/src/include/mb/pg_wchar.h index 64b22e4b0d4..bbce9071dfc 100644 --- a/src/include/mb/pg_wchar.h +++ b/src/include/mb/pg_wchar.h @@ -616,6 +616,12 @@ extern int pg_bind_textdomain_codeset(const char *domainname); extern unsigned char *pg_do_encoding_conversion(unsigned char *src, int len, int src_encoding, int dest_encoding); +extern int pg_do_encoding_conversion_buf(Oid proc, + int src_encoding, + int dest_encoding, + unsigned char *src, int srclen, + unsigned char *dst, int dstlen, + bool noError); extern char *pg_client_to_server(const char *s, int len); extern char *pg_server_to_client(const char *s, int len); @@ -627,18 +633,18 @@ extern void pg_unicode_to_server(pg_wchar c, unsigned char *s); extern unsigned short BIG5toCNS(unsigned short big5, unsigned char *lc); extern unsigned short CNStoBIG5(unsigned short cns, unsigned char lc); -extern void UtfToLocal(const unsigned char *utf, int len, +extern int UtfToLocal(const unsigned char *utf, int len, unsigned char *iso, const pg_mb_radix_tree *map, const pg_utf_to_local_combined *cmap, int cmapsize, utf_local_conversion_func conv_func, - int encoding); -extern void LocalToUtf(const unsigned char *iso, int len, + int encoding, bool noError); +extern int LocalToUtf(const unsigned char *iso, int len, unsigned char *utf, const pg_mb_radix_tree *map, const pg_local_to_utf_combined *cmap, int cmapsize, utf_local_conversion_func conv_func, - int encoding); + int encoding, bool noError); extern bool pg_verifymbstr(const char *mbstr, int len, bool noError); extern bool pg_verify_mbstr(int encoding, const char *mbstr, int len, @@ -656,18 +662,19 @@ extern void report_invalid_encoding(int encoding, const char *mbstr, int len) pg extern void report_untranslatable_char(int src_encoding, int dest_encoding, const char *mbstr, int len) pg_attribute_noreturn(); -extern void local2local(const unsigned char *l, unsigned char *p, int len, - int src_encoding, int dest_encoding, const unsigned char *tab); -extern void latin2mic(const unsigned char *l, unsigned char *p, int len, - int lc, int encoding); -extern void mic2latin(const unsigned char *mic, unsigned char *p, int len, - int lc, int encoding); -extern void latin2mic_with_table(const unsigned char *l, unsigned char *p, +extern int local2local(const unsigned char *l, unsigned char *p, int len, + int src_encoding, int dest_encoding, const unsigned char *tab, + bool noError); +extern int latin2mic(const unsigned char *l, unsigned char *p, int len, + int lc, int encoding, bool noError); +extern int mic2latin(const unsigned char *mic, unsigned char *p, int len, + int lc, int encoding, bool noError); +extern int latin2mic_with_table(const unsigned char *l, unsigned char *p, int len, int lc, int encoding, - const unsigned char *tab); -extern void mic2latin_with_table(const unsigned char *mic, unsigned char *p, + const unsigned char *tab, bool noError); +extern int mic2latin_with_table(const unsigned char *mic, unsigned char *p, int len, int lc, int encoding, - const unsigned char *tab); + const unsigned char *tab, bool noError); #ifdef WIN32 extern WCHAR *pgwin32_message_to_UTF16(const char *str, int len, int *utf16len); diff --git a/src/test/regress/expected/conversion.out b/src/test/regress/expected/conversion.out index 62c10671685..e34ab20974d 100644 --- a/src/test/regress/expected/conversion.out +++ b/src/test/regress/expected/conversion.out @@ -37,3 +37,522 @@ DROP CONVERSION mydef; -- RESET SESSION AUTHORIZATION; DROP USER regress_conversion_user; +-- +-- Test built-in conversion functions. +-- +-- Helper function to test a conversion. Uses the test_enc_conversion function +-- that was created in the create_function_1 test. +create or replace function test_conv( + input IN bytea, + src_encoding IN text, + dst_encoding IN text, + result OUT bytea, + errorat OUT bytea, + error OUT text) +language plpgsql as +$$ +declare + validlen int; +begin + -- First try to perform the conversion with noError = false. If that errors out, + -- capture the error message, and try again with noError = true. The second call + -- should succeed and return the position of the error, return that too. + begin + select * into validlen, result from test_enc_conversion(input, src_encoding, dst_encoding, false); + errorat = NULL; + error := NULL; + exception when others then + error := sqlerrm; + select * into validlen, result from test_enc_conversion(input, src_encoding, dst_encoding, true); + errorat = substr(input, validlen + 1); + end; + return; +end; +$$; +-- +-- UTF-8 +-- +CREATE TABLE utf8_inputs (inbytes bytea, description text); +insert into utf8_inputs values + ('\x666f6f', 'valid, pure ASCII'), + ('\xc3a4c3b6', 'valid, extra latin chars'), + ('\xd184d0bed0be', 'valid, cyrillic'), + ('\x666f6fe8b1a1', 'valid, kanji/Chinese'), + ('\xe382abe3829a', 'valid, two chars that combine to one in EUC_JIS_2004'), + ('\xe382ab', 'only first half of combined char in EUC_JIS_2004'), + ('\xe382abe382', 'incomplete combination when converted EUC_JIS_2004'), + ('\xecbd94eb81bceba6ac', 'valid, Hangul, Korean'), + ('\x666f6fefa8aa', 'valid, needs mapping function to convert to GB18030'), + ('\x66e8b1ff6f6f', 'invalid byte sequence'), + ('\x66006f', 'invalid, NUL byte'), + ('\x666f6fe8b100', 'invalid, NUL byte'), + ('\x666f6fe8b1', 'incomplete character at end'); +-- Test UTF-8 verification +select description, (test_conv(inbytes, 'utf8', 'utf8')).* from utf8_inputs; + description | result | errorat | error +------------------------------------------------------+----------------------+--------------+----------------------------------------------------------- + valid, pure ASCII | \x666f6f | | + valid, extra latin chars | \xc3a4c3b6 | | + valid, cyrillic | \xd184d0bed0be | | + valid, kanji/Chinese | \x666f6fe8b1a1 | | + valid, two chars that combine to one in EUC_JIS_2004 | \xe382abe3829a | | + only first half of combined char in EUC_JIS_2004 | \xe382ab | | + incomplete combination when converted EUC_JIS_2004 | \xe382ab | \xe382 | invalid byte sequence for encoding "UTF8": 0xe3 0x82 + valid, Hangul, Korean | \xecbd94eb81bceba6ac | | + valid, needs mapping function to convert to GB18030 | \x666f6fefa8aa | | + invalid byte sequence | \x66 | \xe8b1ff6f6f | invalid byte sequence for encoding "UTF8": 0xe8 0xb1 0xff + invalid, NUL byte | \x66 | \x006f | invalid byte sequence for encoding "UTF8": 0x00 + invalid, NUL byte | \x666f6f | \xe8b100 | invalid byte sequence for encoding "UTF8": 0xe8 0xb1 0x00 + incomplete character at end | \x666f6f | \xe8b1 | invalid byte sequence for encoding "UTF8": 0xe8 0xb1 +(13 rows) + +-- Test conversions from UTF-8 +select description, inbytes, (test_conv(inbytes, 'utf8', 'euc_jis_2004')).* from utf8_inputs; + description | inbytes | result | errorat | error +------------------------------------------------------+----------------------+----------------+----------------------+------------------------------------------------------------------------------------------------------------- + valid, pure ASCII | \x666f6f | \x666f6f | | + valid, extra latin chars | \xc3a4c3b6 | \xa9daa9ec | | + valid, cyrillic | \xd184d0bed0be | \xa7e6a7e0a7e0 | | + valid, kanji/Chinese | \x666f6fe8b1a1 | \x666f6fbedd | | + valid, two chars that combine to one in EUC_JIS_2004 | \xe382abe3829a | \xa5f7 | | + only first half of combined char in EUC_JIS_2004 | \xe382ab | \xa5ab | | + incomplete combination when converted EUC_JIS_2004 | \xe382abe382 | \x | \xe382abe382 | invalid byte sequence for encoding "UTF8": 0xe3 0x82 + valid, Hangul, Korean | \xecbd94eb81bceba6ac | \x | \xecbd94eb81bceba6ac | character with byte sequence 0xec 0xbd 0x94 in encoding "UTF8" has no equivalent in encoding "EUC_JIS_2004" + valid, needs mapping function to convert to GB18030 | \x666f6fefa8aa | \x666f6f | \xefa8aa | character with byte sequence 0xef 0xa8 0xaa in encoding "UTF8" has no equivalent in encoding "EUC_JIS_2004" + invalid byte sequence | \x66e8b1ff6f6f | \x66 | \xe8b1ff6f6f | invalid byte sequence for encoding "UTF8": 0xe8 0xb1 0xff + invalid, NUL byte | \x66006f | \x66 | \x006f | invalid byte sequence for encoding "UTF8": 0x00 + invalid, NUL byte | \x666f6fe8b100 | \x666f6f | \xe8b100 | invalid byte sequence for encoding "UTF8": 0xe8 0xb1 0x00 + incomplete character at end | \x666f6fe8b1 | \x666f6f | \xe8b1 | invalid byte sequence for encoding "UTF8": 0xe8 0xb1 +(13 rows) + +select description, inbytes, (test_conv(inbytes, 'utf8', 'latin1')).* from utf8_inputs; + description | inbytes | result | errorat | error +------------------------------------------------------+----------------------+----------+----------------------+------------------------------------------------------------------------------------------------------- + valid, pure ASCII | \x666f6f | \x666f6f | | + valid, extra latin chars | \xc3a4c3b6 | \xe4f6 | | + valid, cyrillic | \xd184d0bed0be | \x | \xd184d0bed0be | character with byte sequence 0xd1 0x84 in encoding "UTF8" has no equivalent in encoding "LATIN1" + valid, kanji/Chinese | \x666f6fe8b1a1 | \x666f6f | \xe8b1a1 | character with byte sequence 0xe8 0xb1 0xa1 in encoding "UTF8" has no equivalent in encoding "LATIN1" + valid, two chars that combine to one in EUC_JIS_2004 | \xe382abe3829a | \x | \xe382abe3829a | character with byte sequence 0xe3 0x82 0xab in encoding "UTF8" has no equivalent in encoding "LATIN1" + only first half of combined char in EUC_JIS_2004 | \xe382ab | \x | \xe382ab | character with byte sequence 0xe3 0x82 0xab in encoding "UTF8" has no equivalent in encoding "LATIN1" + incomplete combination when converted EUC_JIS_2004 | \xe382abe382 | \x | \xe382abe382 | character with byte sequence 0xe3 0x82 0xab in encoding "UTF8" has no equivalent in encoding "LATIN1" + valid, Hangul, Korean | \xecbd94eb81bceba6ac | \x | \xecbd94eb81bceba6ac | character with byte sequence 0xec 0xbd 0x94 in encoding "UTF8" has no equivalent in encoding "LATIN1" + valid, needs mapping function to convert to GB18030 | \x666f6fefa8aa | \x666f6f | \xefa8aa | character with byte sequence 0xef 0xa8 0xaa in encoding "UTF8" has no equivalent in encoding "LATIN1" + invalid byte sequence | \x66e8b1ff6f6f | \x66 | \xe8b1ff6f6f | invalid byte sequence for encoding "UTF8": 0xe8 0xb1 0xff + invalid, NUL byte | \x66006f | \x66 | \x006f | invalid byte sequence for encoding "UTF8": 0x00 + invalid, NUL byte | \x666f6fe8b100 | \x666f6f | \xe8b100 | invalid byte sequence for encoding "UTF8": 0xe8 0xb1 0x00 + incomplete character at end | \x666f6fe8b1 | \x666f6f | \xe8b1 | invalid byte sequence for encoding "UTF8": 0xe8 0xb1 +(13 rows) + +select description, inbytes, (test_conv(inbytes, 'utf8', 'latin2')).* from utf8_inputs; + description | inbytes | result | errorat | error +------------------------------------------------------+----------------------+----------+----------------------+------------------------------------------------------------------------------------------------------- + valid, pure ASCII | \x666f6f | \x666f6f | | + valid, extra latin chars | \xc3a4c3b6 | \xe4f6 | | + valid, cyrillic | \xd184d0bed0be | \x | \xd184d0bed0be | character with byte sequence 0xd1 0x84 in encoding "UTF8" has no equivalent in encoding "LATIN2" + valid, kanji/Chinese | \x666f6fe8b1a1 | \x666f6f | \xe8b1a1 | character with byte sequence 0xe8 0xb1 0xa1 in encoding "UTF8" has no equivalent in encoding "LATIN2" + valid, two chars that combine to one in EUC_JIS_2004 | \xe382abe3829a | \x | \xe382abe3829a | character with byte sequence 0xe3 0x82 0xab in encoding "UTF8" has no equivalent in encoding "LATIN2" + only first half of combined char in EUC_JIS_2004 | \xe382ab | \x | \xe382ab | character with byte sequence 0xe3 0x82 0xab in encoding "UTF8" has no equivalent in encoding "LATIN2" + incomplete combination when converted EUC_JIS_2004 | \xe382abe382 | \x | \xe382abe382 | character with byte sequence 0xe3 0x82 0xab in encoding "UTF8" has no equivalent in encoding "LATIN2" + valid, Hangul, Korean | \xecbd94eb81bceba6ac | \x | \xecbd94eb81bceba6ac | character with byte sequence 0xec 0xbd 0x94 in encoding "UTF8" has no equivalent in encoding "LATIN2" + valid, needs mapping function to convert to GB18030 | \x666f6fefa8aa | \x666f6f | \xefa8aa | character with byte sequence 0xef 0xa8 0xaa in encoding "UTF8" has no equivalent in encoding "LATIN2" + invalid byte sequence | \x66e8b1ff6f6f | \x66 | \xe8b1ff6f6f | invalid byte sequence for encoding "UTF8": 0xe8 0xb1 0xff + invalid, NUL byte | \x66006f | \x66 | \x006f | invalid byte sequence for encoding "UTF8": 0x00 + invalid, NUL byte | \x666f6fe8b100 | \x666f6f | \xe8b100 | invalid byte sequence for encoding "UTF8": 0xe8 0xb1 0x00 + incomplete character at end | \x666f6fe8b1 | \x666f6f | \xe8b1 | invalid byte sequence for encoding "UTF8": 0xe8 0xb1 +(13 rows) + +select description, inbytes, (test_conv(inbytes, 'utf8', 'latin5')).* from utf8_inputs; + description | inbytes | result | errorat | error +------------------------------------------------------+----------------------+----------+----------------------+------------------------------------------------------------------------------------------------------- + valid, pure ASCII | \x666f6f | \x666f6f | | + valid, extra latin chars | \xc3a4c3b6 | \xe4f6 | | + valid, cyrillic | \xd184d0bed0be | \x | \xd184d0bed0be | character with byte sequence 0xd1 0x84 in encoding "UTF8" has no equivalent in encoding "LATIN5" + valid, kanji/Chinese | \x666f6fe8b1a1 | \x666f6f | \xe8b1a1 | character with byte sequence 0xe8 0xb1 0xa1 in encoding "UTF8" has no equivalent in encoding "LATIN5" + valid, two chars that combine to one in EUC_JIS_2004 | \xe382abe3829a | \x | \xe382abe3829a | character with byte sequence 0xe3 0x82 0xab in encoding "UTF8" has no equivalent in encoding "LATIN5" + only first half of combined char in EUC_JIS_2004 | \xe382ab | \x | \xe382ab | character with byte sequence 0xe3 0x82 0xab in encoding "UTF8" has no equivalent in encoding "LATIN5" + incomplete combination when converted EUC_JIS_2004 | \xe382abe382 | \x | \xe382abe382 | character with byte sequence 0xe3 0x82 0xab in encoding "UTF8" has no equivalent in encoding "LATIN5" + valid, Hangul, Korean | \xecbd94eb81bceba6ac | \x | \xecbd94eb81bceba6ac | character with byte sequence 0xec 0xbd 0x94 in encoding "UTF8" has no equivalent in encoding "LATIN5" + valid, needs mapping function to convert to GB18030 | \x666f6fefa8aa | \x666f6f | \xefa8aa | character with byte sequence 0xef 0xa8 0xaa in encoding "UTF8" has no equivalent in encoding "LATIN5" + invalid byte sequence | \x66e8b1ff6f6f | \x66 | \xe8b1ff6f6f | invalid byte sequence for encoding "UTF8": 0xe8 0xb1 0xff + invalid, NUL byte | \x66006f | \x66 | \x006f | invalid byte sequence for encoding "UTF8": 0x00 + invalid, NUL byte | \x666f6fe8b100 | \x666f6f | \xe8b100 | invalid byte sequence for encoding "UTF8": 0xe8 0xb1 0x00 + incomplete character at end | \x666f6fe8b1 | \x666f6f | \xe8b1 | invalid byte sequence for encoding "UTF8": 0xe8 0xb1 +(13 rows) + +select description, inbytes, (test_conv(inbytes, 'utf8', 'koi8r')).* from utf8_inputs; + description | inbytes | result | errorat | error +------------------------------------------------------+----------------------+----------+----------------------+------------------------------------------------------------------------------------------------------ + valid, pure ASCII | \x666f6f | \x666f6f | | + valid, extra latin chars | \xc3a4c3b6 | \x | \xc3a4c3b6 | character with byte sequence 0xc3 0xa4 in encoding "UTF8" has no equivalent in encoding "KOI8R" + valid, cyrillic | \xd184d0bed0be | \xc6cfcf | | + valid, kanji/Chinese | \x666f6fe8b1a1 | \x666f6f | \xe8b1a1 | character with byte sequence 0xe8 0xb1 0xa1 in encoding "UTF8" has no equivalent in encoding "KOI8R" + valid, two chars that combine to one in EUC_JIS_2004 | \xe382abe3829a | \x | \xe382abe3829a | character with byte sequence 0xe3 0x82 0xab in encoding "UTF8" has no equivalent in encoding "KOI8R" + only first half of combined char in EUC_JIS_2004 | \xe382ab | \x | \xe382ab | character with byte sequence 0xe3 0x82 0xab in encoding "UTF8" has no equivalent in encoding "KOI8R" + incomplete combination when converted EUC_JIS_2004 | \xe382abe382 | \x | \xe382abe382 | character with byte sequence 0xe3 0x82 0xab in encoding "UTF8" has no equivalent in encoding "KOI8R" + valid, Hangul, Korean | \xecbd94eb81bceba6ac | \x | \xecbd94eb81bceba6ac | character with byte sequence 0xec 0xbd 0x94 in encoding "UTF8" has no equivalent in encoding "KOI8R" + valid, needs mapping function to convert to GB18030 | \x666f6fefa8aa | \x666f6f | \xefa8aa | character with byte sequence 0xef 0xa8 0xaa in encoding "UTF8" has no equivalent in encoding "KOI8R" + invalid byte sequence | \x66e8b1ff6f6f | \x66 | \xe8b1ff6f6f | invalid byte sequence for encoding "UTF8": 0xe8 0xb1 0xff + invalid, NUL byte | \x66006f | \x66 | \x006f | invalid byte sequence for encoding "UTF8": 0x00 + invalid, NUL byte | \x666f6fe8b100 | \x666f6f | \xe8b100 | invalid byte sequence for encoding "UTF8": 0xe8 0xb1 0x00 + incomplete character at end | \x666f6fe8b1 | \x666f6f | \xe8b1 | invalid byte sequence for encoding "UTF8": 0xe8 0xb1 +(13 rows) + +select description, inbytes, (test_conv(inbytes, 'utf8', 'gb18030')).* from utf8_inputs; + description | inbytes | result | errorat | error +------------------------------------------------------+----------------------+----------------------------+--------------+----------------------------------------------------------- + valid, pure ASCII | \x666f6f | \x666f6f | | + valid, extra latin chars | \xc3a4c3b6 | \x81308a3181308b32 | | + valid, cyrillic | \xd184d0bed0be | \xa7e6a7e0a7e0 | | + valid, kanji/Chinese | \x666f6fe8b1a1 | \x666f6fcff3 | | + valid, two chars that combine to one in EUC_JIS_2004 | \xe382abe3829a | \xa5ab8139a732 | | + only first half of combined char in EUC_JIS_2004 | \xe382ab | \xa5ab | | + incomplete combination when converted EUC_JIS_2004 | \xe382abe382 | \xa5ab | \xe382 | invalid byte sequence for encoding "UTF8": 0xe3 0x82 + valid, Hangul, Korean | \xecbd94eb81bceba6ac | \x8334e5398238c4338330b335 | | + valid, needs mapping function to convert to GB18030 | \x666f6fefa8aa | \x666f6f84309c38 | | + invalid byte sequence | \x66e8b1ff6f6f | \x66 | \xe8b1ff6f6f | invalid byte sequence for encoding "UTF8": 0xe8 0xb1 0xff + invalid, NUL byte | \x66006f | \x66 | \x006f | invalid byte sequence for encoding "UTF8": 0x00 + invalid, NUL byte | \x666f6fe8b100 | \x666f6f | \xe8b100 | invalid byte sequence for encoding "UTF8": 0xe8 0xb1 0x00 + incomplete character at end | \x666f6fe8b1 | \x666f6f | \xe8b1 | invalid byte sequence for encoding "UTF8": 0xe8 0xb1 +(13 rows) + +-- +-- EUC_JIS_2004 +-- +CREATE TABLE euc_jis_2004_inputs (inbytes bytea, description text); +insert into euc_jis_2004_inputs values + ('\x666f6f', 'valid, pure ASCII'), + ('\x666f6fbedd', 'valid'), + ('\xa5f7', 'valid, translates to two UTF-8 chars '), + ('\xbeddbe', 'incomplete char '), + ('\x666f6f00bedd', 'invalid, NUL byte'), + ('\x666f6fbe00dd', 'invalid, NUL byte'), + ('\x666f6fbedd00', 'invalid, NUL byte'), + ('\xbe04', 'invalid byte sequence'); +-- Test EUC_JIS_2004 verification +select description, inbytes, (test_conv(inbytes, 'euc_jis_2004', 'euc_jis_2004')).* from euc_jis_2004_inputs; + description | inbytes | result | errorat | error +---------------------------------------+----------------+--------------+----------+-------------------------------------------------------------- + valid, pure ASCII | \x666f6f | \x666f6f | | + valid | \x666f6fbedd | \x666f6fbedd | | + valid, translates to two UTF-8 chars | \xa5f7 | \xa5f7 | | + incomplete char | \xbeddbe | \xbedd | \xbe | invalid byte sequence for encoding "EUC_JIS_2004": 0xbe + invalid, NUL byte | \x666f6f00bedd | \x666f6f | \x00bedd | invalid byte sequence for encoding "EUC_JIS_2004": 0x00 + invalid, NUL byte | \x666f6fbe00dd | \x666f6f | \xbe00dd | invalid byte sequence for encoding "EUC_JIS_2004": 0xbe 0x00 + invalid, NUL byte | \x666f6fbedd00 | \x666f6fbedd | \x00 | invalid byte sequence for encoding "EUC_JIS_2004": 0x00 + invalid byte sequence | \xbe04 | \x | \xbe04 | invalid byte sequence for encoding "EUC_JIS_2004": 0xbe 0x04 +(8 rows) + +-- Test conversions from EUC_JIS_2004 +select description, inbytes, (test_conv(inbytes, 'euc_jis_2004', 'utf8')).* from euc_jis_2004_inputs; + description | inbytes | result | errorat | error +---------------------------------------+----------------+----------------+----------+-------------------------------------------------------------- + valid, pure ASCII | \x666f6f | \x666f6f | | + valid | \x666f6fbedd | \x666f6fe8b1a1 | | + valid, translates to two UTF-8 chars | \xa5f7 | \xe382abe3829a | | + incomplete char | \xbeddbe | \xe8b1a1 | \xbe | invalid byte sequence for encoding "EUC_JIS_2004": 0xbe + invalid, NUL byte | \x666f6f00bedd | \x666f6f | \x00bedd | invalid byte sequence for encoding "EUC_JIS_2004": 0x00 + invalid, NUL byte | \x666f6fbe00dd | \x666f6f | \xbe00dd | invalid byte sequence for encoding "EUC_JIS_2004": 0xbe 0x00 + invalid, NUL byte | \x666f6fbedd00 | \x666f6fe8b1a1 | \x00 | invalid byte sequence for encoding "EUC_JIS_2004": 0x00 + invalid byte sequence | \xbe04 | \x | \xbe04 | invalid byte sequence for encoding "EUC_JIS_2004": 0xbe 0x04 +(8 rows) + +-- +-- SHIFT-JIS-2004 +-- +CREATE TABLE shiftjis2004_inputs (inbytes bytea, description text); +insert into shiftjis2004_inputs values + ('\x666f6f', 'valid, pure ASCII'), + ('\x666f6f8fdb', 'valid'), + ('\x666f6f81c0', 'valid, no translation to UTF-8'), + ('\x666f6f82f5', 'valid, translates to two UTF-8 chars '), + ('\x666f6f8fdb8f', 'incomplete char '), + ('\x666f6f820a', 'incomplete char, followed by newline '), + ('\x666f6f008fdb', 'invalid, NUL byte'), + ('\x666f6f8f00db', 'invalid, NUL byte'), + ('\x666f6f8fdb00', 'invalid, NUL byte'); +-- Test SHIFT-JIS-2004 verification +select description, inbytes, (test_conv(inbytes, 'shiftjis2004', 'shiftjis2004')).* from shiftjis2004_inputs; + description | inbytes | result | errorat | error +---------------------------------------+----------------+--------------+----------+---------------------------------------------------------------- + valid, pure ASCII | \x666f6f | \x666f6f | | + valid | \x666f6f8fdb | \x666f6f8fdb | | + valid, no translation to UTF-8 | \x666f6f81c0 | \x666f6f81c0 | | + valid, translates to two UTF-8 chars | \x666f6f82f5 | \x666f6f82f5 | | + incomplete char | \x666f6f8fdb8f | \x666f6f8fdb | \x8f | invalid byte sequence for encoding "SHIFT_JIS_2004": 0x8f + incomplete char, followed by newline | \x666f6f820a | \x666f6f | \x820a | invalid byte sequence for encoding "SHIFT_JIS_2004": 0x82 0x0a + invalid, NUL byte | \x666f6f008fdb | \x666f6f | \x008fdb | invalid byte sequence for encoding "SHIFT_JIS_2004": 0x00 + invalid, NUL byte | \x666f6f8f00db | \x666f6f | \x8f00db | invalid byte sequence for encoding "SHIFT_JIS_2004": 0x8f 0x00 + invalid, NUL byte | \x666f6f8fdb00 | \x666f6f8fdb | \x00 | invalid byte sequence for encoding "SHIFT_JIS_2004": 0x00 +(9 rows) + +-- Test conversions from SHIFT-JIS-2004 +select description, inbytes, (test_conv(inbytes, 'shiftjis2004', 'utf8')).* from shiftjis2004_inputs; + description | inbytes | result | errorat | error +---------------------------------------+----------------+----------------------+----------+---------------------------------------------------------------- + valid, pure ASCII | \x666f6f | \x666f6f | | + valid | \x666f6f8fdb | \x666f6fe8b1a1 | | + valid, no translation to UTF-8 | \x666f6f81c0 | \x666f6fe28a84 | | + valid, translates to two UTF-8 chars | \x666f6f82f5 | \x666f6fe3818be3829a | | + incomplete char | \x666f6f8fdb8f | \x666f6fe8b1a1 | \x8f | invalid byte sequence for encoding "SHIFT_JIS_2004": 0x8f + incomplete char, followed by newline | \x666f6f820a | \x666f6f | \x820a | invalid byte sequence for encoding "SHIFT_JIS_2004": 0x82 0x0a + invalid, NUL byte | \x666f6f008fdb | \x666f6f | \x008fdb | invalid byte sequence for encoding "SHIFT_JIS_2004": 0x00 + invalid, NUL byte | \x666f6f8f00db | \x666f6f | \x8f00db | invalid byte sequence for encoding "SHIFT_JIS_2004": 0x8f 0x00 + invalid, NUL byte | \x666f6f8fdb00 | \x666f6fe8b1a1 | \x00 | invalid byte sequence for encoding "SHIFT_JIS_2004": 0x00 +(9 rows) + +select description, inbytes, (test_conv(inbytes, 'shiftjis2004', 'euc_jis_2004')).* from shiftjis2004_inputs; + description | inbytes | result | errorat | error +---------------------------------------+----------------+--------------+----------+---------------------------------------------------------------- + valid, pure ASCII | \x666f6f | \x666f6f | | + valid | \x666f6f8fdb | \x666f6fbedd | | + valid, no translation to UTF-8 | \x666f6f81c0 | \x666f6fa2c2 | | + valid, translates to two UTF-8 chars | \x666f6f82f5 | \x666f6fa4f7 | | + incomplete char | \x666f6f8fdb8f | \x666f6fbedd | \x8f | invalid byte sequence for encoding "SHIFT_JIS_2004": 0x8f + incomplete char, followed by newline | \x666f6f820a | \x666f6f | \x820a | invalid byte sequence for encoding "SHIFT_JIS_2004": 0x82 0x0a + invalid, NUL byte | \x666f6f008fdb | \x666f6f | \x008fdb | invalid byte sequence for encoding "SHIFT_JIS_2004": 0x00 + invalid, NUL byte | \x666f6f8f00db | \x666f6f | \x8f00db | invalid byte sequence for encoding "SHIFT_JIS_2004": 0x8f 0x00 + invalid, NUL byte | \x666f6f8fdb00 | \x666f6fbedd | \x00 | invalid byte sequence for encoding "SHIFT_JIS_2004": 0x00 +(9 rows) + +-- +-- GB18030 +-- +CREATE TABLE gb18030_inputs (inbytes bytea, description text); +insert into gb18030_inputs values + ('\x666f6f', 'valid, pure ASCII'), + ('\x666f6fcff3', 'valid'), + ('\x666f6f8431a530', 'valid, no translation to UTF-8'), + ('\x666f6f84309c38', 'valid, translates to UTF-8 by mapping function'), + ('\x666f6f84309c', 'incomplete char '), + ('\x666f6f84309c0a', 'incomplete char, followed by newline '), + ('\x666f6f84309c3800', 'invalid, NUL byte'), + ('\x666f6f84309c0038', 'invalid, NUL byte'); +-- Test GB18030 verification +select description, inbytes, (test_conv(inbytes, 'gb18030', 'gb18030')).* from gb18030_inputs; + description | inbytes | result | errorat | error +------------------------------------------------+--------------------+------------------+--------------+------------------------------------------------------------------- + valid, pure ASCII | \x666f6f | \x666f6f | | + valid | \x666f6fcff3 | \x666f6fcff3 | | + valid, no translation to UTF-8 | \x666f6f8431a530 | \x666f6f8431a530 | | + valid, translates to UTF-8 by mapping function | \x666f6f84309c38 | \x666f6f84309c38 | | + incomplete char | \x666f6f84309c | \x666f6f | \x84309c | invalid byte sequence for encoding "GB18030": 0x84 0x30 0x9c + incomplete char, followed by newline | \x666f6f84309c0a | \x666f6f | \x84309c0a | invalid byte sequence for encoding "GB18030": 0x84 0x30 0x9c 0x0a + invalid, NUL byte | \x666f6f84309c3800 | \x666f6f84309c38 | \x00 | invalid byte sequence for encoding "GB18030": 0x00 + invalid, NUL byte | \x666f6f84309c0038 | \x666f6f | \x84309c0038 | invalid byte sequence for encoding "GB18030": 0x84 0x30 0x9c 0x00 +(8 rows) + +-- Test conversions from GB18030 +select description, inbytes, (test_conv(inbytes, 'gb18030', 'utf8')).* from gb18030_inputs; + description | inbytes | result | errorat | error +------------------------------------------------+--------------------+----------------+--------------+------------------------------------------------------------------------------------------------------------- + valid, pure ASCII | \x666f6f | \x666f6f | | + valid | \x666f6fcff3 | \x666f6fe8b1a1 | | + valid, no translation to UTF-8 | \x666f6f8431a530 | \x666f6f | \x8431a530 | character with byte sequence 0x84 0x31 0xa5 0x30 in encoding "GB18030" has no equivalent in encoding "UTF8" + valid, translates to UTF-8 by mapping function | \x666f6f84309c38 | \x666f6fefa8aa | | + incomplete char | \x666f6f84309c | \x666f6f | \x84309c | invalid byte sequence for encoding "GB18030": 0x84 0x30 0x9c + incomplete char, followed by newline | \x666f6f84309c0a | \x666f6f | \x84309c0a | invalid byte sequence for encoding "GB18030": 0x84 0x30 0x9c 0x0a + invalid, NUL byte | \x666f6f84309c3800 | \x666f6fefa8aa | \x00 | invalid byte sequence for encoding "GB18030": 0x00 + invalid, NUL byte | \x666f6f84309c0038 | \x666f6f | \x84309c0038 | invalid byte sequence for encoding "GB18030": 0x84 0x30 0x9c 0x00 +(8 rows) + +-- +-- ISO-8859-5 +-- +CREATE TABLE iso8859_5_inputs (inbytes bytea, description text); +insert into iso8859_5_inputs values + ('\x666f6f', 'valid, pure ASCII'), + ('\xe4dede', 'valid'), + ('\x00', 'invalid, NUL byte'), + ('\xe400dede', 'invalid, NUL byte'), + ('\xe4dede00', 'invalid, NUL byte'); +-- Test ISO-8859-5 verification +select description, inbytes, (test_conv(inbytes, 'iso8859-5', 'iso8859-5')).* from iso8859_5_inputs; + description | inbytes | result | errorat | error +-------------------+------------+----------+----------+------------------------------------------------------- + valid, pure ASCII | \x666f6f | \x666f6f | | + valid | \xe4dede | \xe4dede | | + invalid, NUL byte | \x00 | \x | \x00 | invalid byte sequence for encoding "ISO_8859_5": 0x00 + invalid, NUL byte | \xe400dede | \xe4 | \x00dede | invalid byte sequence for encoding "ISO_8859_5": 0x00 + invalid, NUL byte | \xe4dede00 | \xe4dede | \x00 | invalid byte sequence for encoding "ISO_8859_5": 0x00 +(5 rows) + +-- Test conversions from ISO-8859-5 +select description, inbytes, (test_conv(inbytes, 'iso8859-5', 'utf8')).* from iso8859_5_inputs; + description | inbytes | result | errorat | error +-------------------+------------+----------------+----------+------------------------------------------------------- + valid, pure ASCII | \x666f6f | \x666f6f | | + valid | \xe4dede | \xd184d0bed0be | | + invalid, NUL byte | \x00 | \x | \x00 | invalid byte sequence for encoding "ISO_8859_5": 0x00 + invalid, NUL byte | \xe400dede | \xd184 | \x00dede | invalid byte sequence for encoding "ISO_8859_5": 0x00 + invalid, NUL byte | \xe4dede00 | \xd184d0bed0be | \x00 | invalid byte sequence for encoding "ISO_8859_5": 0x00 +(5 rows) + +select description, inbytes, (test_conv(inbytes, 'iso8859-5', 'koi8r')).* from iso8859_5_inputs; + description | inbytes | result | errorat | error +-------------------+------------+----------+----------+------------------------------------------------------- + valid, pure ASCII | \x666f6f | \x666f6f | | + valid | \xe4dede | \xc6cfcf | | + invalid, NUL byte | \x00 | \x | \x00 | invalid byte sequence for encoding "ISO_8859_5": 0x00 + invalid, NUL byte | \xe400dede | \xc6 | \x00dede | invalid byte sequence for encoding "ISO_8859_5": 0x00 + invalid, NUL byte | \xe4dede00 | \xc6cfcf | \x00 | invalid byte sequence for encoding "ISO_8859_5": 0x00 +(5 rows) + +select description, inbytes, (test_conv(inbytes, 'iso8859_5', 'mule_internal')).* from iso8859_5_inputs; + description | inbytes | result | errorat | error +-------------------+------------+----------------+----------+------------------------------------------------------- + valid, pure ASCII | \x666f6f | \x666f6f | | + valid | \xe4dede | \x8bc68bcf8bcf | | + invalid, NUL byte | \x00 | \x | \x00 | invalid byte sequence for encoding "ISO_8859_5": 0x00 + invalid, NUL byte | \xe400dede | \x8bc6 | \x00dede | invalid byte sequence for encoding "ISO_8859_5": 0x00 + invalid, NUL byte | \xe4dede00 | \x8bc68bcf8bcf | \x00 | invalid byte sequence for encoding "ISO_8859_5": 0x00 +(5 rows) + +-- +-- Big5 +-- +CREATE TABLE big5_inputs (inbytes bytea, description text); +insert into big5_inputs values + ('\x666f6f', 'valid, pure ASCII'), + ('\x666f6fb648', 'valid'), + ('\x666f6fa27f', 'valid, no translation to UTF-8'), + ('\x666f6fb60048', 'invalid, NUL byte'), + ('\x666f6fb64800', 'invalid, NUL byte'); +-- Test Big5 verification +select description, inbytes, (test_conv(inbytes, 'big5', 'big5')).* from big5_inputs; + description | inbytes | result | errorat | error +--------------------------------+----------------+--------------+----------+------------------------------------------------------ + valid, pure ASCII | \x666f6f | \x666f6f | | + valid | \x666f6fb648 | \x666f6fb648 | | + valid, no translation to UTF-8 | \x666f6fa27f | \x666f6fa27f | | + invalid, NUL byte | \x666f6fb60048 | \x666f6f | \xb60048 | invalid byte sequence for encoding "BIG5": 0xb6 0x00 + invalid, NUL byte | \x666f6fb64800 | \x666f6fb648 | \x00 | invalid byte sequence for encoding "BIG5": 0x00 +(5 rows) + +-- Test conversions from Big5 +select description, inbytes, (test_conv(inbytes, 'big5', 'utf8')).* from big5_inputs; + description | inbytes | result | errorat | error +--------------------------------+----------------+----------------+----------+------------------------------------------------------------------------------------------------ + valid, pure ASCII | \x666f6f | \x666f6f | | + valid | \x666f6fb648 | \x666f6fe8b1a1 | | + valid, no translation to UTF-8 | \x666f6fa27f | \x666f6f | \xa27f | character with byte sequence 0xa2 0x7f in encoding "BIG5" has no equivalent in encoding "UTF8" + invalid, NUL byte | \x666f6fb60048 | \x666f6f | \xb60048 | invalid byte sequence for encoding "BIG5": 0xb6 0x00 + invalid, NUL byte | \x666f6fb64800 | \x666f6fe8b1a1 | \x00 | invalid byte sequence for encoding "BIG5": 0x00 +(5 rows) + +select description, inbytes, (test_conv(inbytes, 'big5', 'mule_internal')).* from big5_inputs; + description | inbytes | result | errorat | error +--------------------------------+----------------+----------------+----------+------------------------------------------------------ + valid, pure ASCII | \x666f6f | \x666f6f | | + valid | \x666f6fb648 | \x666f6f95e2af | | + valid, no translation to UTF-8 | \x666f6fa27f | \x666f6f95a3c1 | | + invalid, NUL byte | \x666f6fb60048 | \x666f6f | \xb60048 | invalid byte sequence for encoding "BIG5": 0xb6 0x00 + invalid, NUL byte | \x666f6fb64800 | \x666f6f95e2af | \x00 | invalid byte sequence for encoding "BIG5": 0x00 +(5 rows) + +-- +-- MULE_INTERNAL +-- +CREATE TABLE mic_inputs (inbytes bytea, description text); +insert into mic_inputs values + ('\x666f6f', 'valid, pure ASCII'), + ('\x8bc68bcf8bcf', 'valid (in KOI8R)'), + ('\x8bc68bcf8b', 'invalid,incomplete char'), + ('\x92bedd', 'valid (in SHIFT_JIS)'), + ('\x92be', 'invalid, incomplete char)'), + ('\x666f6f95a3c1', 'valid (in Big5)'), + ('\x666f6f95a3', 'invalid, incomplete char'), + ('\x9200bedd', 'invalid, NUL byte'), + ('\x92bedd00', 'invalid, NUL byte'), + ('\x8b00c68bcf8bcf', 'invalid, NUL byte'); +-- Test MULE_INTERNAL verification +select description, inbytes, (test_conv(inbytes, 'mule_internal', 'mule_internal')).* from mic_inputs; + description | inbytes | result | errorat | error +---------------------------+------------------+----------------+------------------+-------------------------------------------------------------------- + valid, pure ASCII | \x666f6f | \x666f6f | | + valid (in KOI8R) | \x8bc68bcf8bcf | \x8bc68bcf8bcf | | + invalid,incomplete char | \x8bc68bcf8b | \x8bc68bcf | \x8b | invalid byte sequence for encoding "MULE_INTERNAL": 0x8b + valid (in SHIFT_JIS) | \x92bedd | \x92bedd | | + invalid, incomplete char) | \x92be | \x | \x92be | invalid byte sequence for encoding "MULE_INTERNAL": 0x92 0xbe + valid (in Big5) | \x666f6f95a3c1 | \x666f6f95a3c1 | | + invalid, incomplete char | \x666f6f95a3 | \x666f6f | \x95a3 | invalid byte sequence for encoding "MULE_INTERNAL": 0x95 0xa3 + invalid, NUL byte | \x9200bedd | \x | \x9200bedd | invalid byte sequence for encoding "MULE_INTERNAL": 0x92 0x00 0xbe + invalid, NUL byte | \x92bedd00 | \x92bedd | \x00 | invalid byte sequence for encoding "MULE_INTERNAL": 0x00 + invalid, NUL byte | \x8b00c68bcf8bcf | \x | \x8b00c68bcf8bcf | invalid byte sequence for encoding "MULE_INTERNAL": 0x8b 0x00 +(10 rows) + +-- Test conversions from MULE_INTERNAL +select description, inbytes, (test_conv(inbytes, 'mule_internal', 'koi8r')).* from mic_inputs; + description | inbytes | result | errorat | error +---------------------------+------------------+----------+------------------+--------------------------------------------------------------------------------------------------------------- + valid, pure ASCII | \x666f6f | \x666f6f | | + valid (in KOI8R) | \x8bc68bcf8bcf | \xc6cfcf | | + invalid,incomplete char | \x8bc68bcf8b | \xc6cf | \x8b | invalid byte sequence for encoding "MULE_INTERNAL": 0x8b + valid (in SHIFT_JIS) | \x92bedd | \x | \x92bedd | character with byte sequence 0x92 0xbe 0xdd in encoding "MULE_INTERNAL" has no equivalent in encoding "KOI8R" + invalid, incomplete char) | \x92be | \x | \x92be | invalid byte sequence for encoding "MULE_INTERNAL": 0x92 0xbe + valid (in Big5) | \x666f6f95a3c1 | \x666f6f | \x95a3c1 | character with byte sequence 0x95 0xa3 0xc1 in encoding "MULE_INTERNAL" has no equivalent in encoding "KOI8R" + invalid, incomplete char | \x666f6f95a3 | \x666f6f | \x95a3 | invalid byte sequence for encoding "MULE_INTERNAL": 0x95 0xa3 + invalid, NUL byte | \x9200bedd | \x | \x9200bedd | character with byte sequence 0x92 0x00 0xbe in encoding "MULE_INTERNAL" has no equivalent in encoding "KOI8R" + invalid, NUL byte | \x92bedd00 | \x | \x92bedd00 | character with byte sequence 0x92 0xbe 0xdd in encoding "MULE_INTERNAL" has no equivalent in encoding "KOI8R" + invalid, NUL byte | \x8b00c68bcf8bcf | \x | \x8b00c68bcf8bcf | character with byte sequence 0x8b 0x00 in encoding "MULE_INTERNAL" has no equivalent in encoding "KOI8R" +(10 rows) + +select description, inbytes, (test_conv(inbytes, 'mule_internal', 'iso8859-5')).* from mic_inputs; + description | inbytes | result | errorat | error +---------------------------+------------------+----------+------------------+-------------------------------------------------------------------------------------------------------------------- + valid, pure ASCII | \x666f6f | \x666f6f | | + valid (in KOI8R) | \x8bc68bcf8bcf | \xe4dede | | + invalid,incomplete char | \x8bc68bcf8b | \xe4de | \x8b | invalid byte sequence for encoding "MULE_INTERNAL": 0x8b + valid (in SHIFT_JIS) | \x92bedd | \x | \x92bedd | character with byte sequence 0x92 0xbe 0xdd in encoding "MULE_INTERNAL" has no equivalent in encoding "ISO_8859_5" + invalid, incomplete char) | \x92be | \x | \x92be | invalid byte sequence for encoding "MULE_INTERNAL": 0x92 0xbe + valid (in Big5) | \x666f6f95a3c1 | \x666f6f | \x95a3c1 | character with byte sequence 0x95 0xa3 0xc1 in encoding "MULE_INTERNAL" has no equivalent in encoding "ISO_8859_5" + invalid, incomplete char | \x666f6f95a3 | \x666f6f | \x95a3 | invalid byte sequence for encoding "MULE_INTERNAL": 0x95 0xa3 + invalid, NUL byte | \x9200bedd | \x | \x9200bedd | character with byte sequence 0x92 0x00 0xbe in encoding "MULE_INTERNAL" has no equivalent in encoding "ISO_8859_5" + invalid, NUL byte | \x92bedd00 | \x | \x92bedd00 | character with byte sequence 0x92 0xbe 0xdd in encoding "MULE_INTERNAL" has no equivalent in encoding "ISO_8859_5" + invalid, NUL byte | \x8b00c68bcf8bcf | \x | \x8b00c68bcf8bcf | character with byte sequence 0x8b 0x00 in encoding "MULE_INTERNAL" has no equivalent in encoding "ISO_8859_5" +(10 rows) + +select description, inbytes, (test_conv(inbytes, 'mule_internal', 'sjis')).* from mic_inputs; + description | inbytes | result | errorat | error +---------------------------+------------------+----------+------------------+-------------------------------------------------------------------------------------------------------------- + valid, pure ASCII | \x666f6f | \x666f6f | | + valid (in KOI8R) | \x8bc68bcf8bcf | \x | \x8bc68bcf8bcf | character with byte sequence 0x8b 0xc6 in encoding "MULE_INTERNAL" has no equivalent in encoding "SJIS" + invalid,incomplete char | \x8bc68bcf8b | \x | \x8bc68bcf8b | character with byte sequence 0x8b 0xc6 in encoding "MULE_INTERNAL" has no equivalent in encoding "SJIS" + valid (in SHIFT_JIS) | \x92bedd | \x8fdb | | + invalid, incomplete char) | \x92be | \x | \x92be | invalid byte sequence for encoding "MULE_INTERNAL": 0x92 0xbe + valid (in Big5) | \x666f6f95a3c1 | \x666f6f | \x95a3c1 | character with byte sequence 0x95 0xa3 0xc1 in encoding "MULE_INTERNAL" has no equivalent in encoding "SJIS" + invalid, incomplete char | \x666f6f95a3 | \x666f6f | \x95a3 | invalid byte sequence for encoding "MULE_INTERNAL": 0x95 0xa3 + invalid, NUL byte | \x9200bedd | \x | \x9200bedd | invalid byte sequence for encoding "MULE_INTERNAL": 0x92 0x00 0xbe + invalid, NUL byte | \x92bedd00 | \x8fdb | \x00 | invalid byte sequence for encoding "MULE_INTERNAL": 0x00 + invalid, NUL byte | \x8b00c68bcf8bcf | \x | \x8b00c68bcf8bcf | invalid byte sequence for encoding "MULE_INTERNAL": 0x8b 0x00 +(10 rows) + +select description, inbytes, (test_conv(inbytes, 'mule_internal', 'big5')).* from mic_inputs; + description | inbytes | result | errorat | error +---------------------------+------------------+--------------+------------------+-------------------------------------------------------------------------------------------------------------- + valid, pure ASCII | \x666f6f | \x666f6f | | + valid (in KOI8R) | \x8bc68bcf8bcf | \x | \x8bc68bcf8bcf | character with byte sequence 0x8b 0xc6 in encoding "MULE_INTERNAL" has no equivalent in encoding "BIG5" + invalid,incomplete char | \x8bc68bcf8b | \x | \x8bc68bcf8b | character with byte sequence 0x8b 0xc6 in encoding "MULE_INTERNAL" has no equivalent in encoding "BIG5" + valid (in SHIFT_JIS) | \x92bedd | \x | \x92bedd | character with byte sequence 0x92 0xbe 0xdd in encoding "MULE_INTERNAL" has no equivalent in encoding "BIG5" + invalid, incomplete char) | \x92be | \x | \x92be | invalid byte sequence for encoding "MULE_INTERNAL": 0x92 0xbe + valid (in Big5) | \x666f6f95a3c1 | \x666f6fa2a1 | | + invalid, incomplete char | \x666f6f95a3 | \x666f6f | \x95a3 | invalid byte sequence for encoding "MULE_INTERNAL": 0x95 0xa3 + invalid, NUL byte | \x9200bedd | \x | \x9200bedd | invalid byte sequence for encoding "MULE_INTERNAL": 0x92 0x00 0xbe + invalid, NUL byte | \x92bedd00 | \x | \x92bedd00 | character with byte sequence 0x92 0xbe 0xdd in encoding "MULE_INTERNAL" has no equivalent in encoding "BIG5" + invalid, NUL byte | \x8b00c68bcf8bcf | \x | \x8b00c68bcf8bcf | invalid byte sequence for encoding "MULE_INTERNAL": 0x8b 0x00 +(10 rows) + +select description, inbytes, (test_conv(inbytes, 'mule_internal', 'euc_jp')).* from mic_inputs; + description | inbytes | result | errorat | error +---------------------------+------------------+----------+------------------+---------------------------------------------------------------------------------------------------------------- + valid, pure ASCII | \x666f6f | \x666f6f | | + valid (in KOI8R) | \x8bc68bcf8bcf | \x | \x8bc68bcf8bcf | character with byte sequence 0x8b 0xc6 in encoding "MULE_INTERNAL" has no equivalent in encoding "EUC_JP" + invalid,incomplete char | \x8bc68bcf8b | \x | \x8bc68bcf8b | character with byte sequence 0x8b 0xc6 in encoding "MULE_INTERNAL" has no equivalent in encoding "EUC_JP" + valid (in SHIFT_JIS) | \x92bedd | \xbedd | | + invalid, incomplete char) | \x92be | \x | \x92be | invalid byte sequence for encoding "MULE_INTERNAL": 0x92 0xbe + valid (in Big5) | \x666f6f95a3c1 | \x666f6f | \x95a3c1 | character with byte sequence 0x95 0xa3 0xc1 in encoding "MULE_INTERNAL" has no equivalent in encoding "EUC_JP" + invalid, incomplete char | \x666f6f95a3 | \x666f6f | \x95a3 | invalid byte sequence for encoding "MULE_INTERNAL": 0x95 0xa3 + invalid, NUL byte | \x9200bedd | \x | \x9200bedd | invalid byte sequence for encoding "MULE_INTERNAL": 0x92 0x00 0xbe + invalid, NUL byte | \x92bedd00 | \xbedd | \x00 | invalid byte sequence for encoding "MULE_INTERNAL": 0x00 + invalid, NUL byte | \x8b00c68bcf8bcf | \x | \x8b00c68bcf8bcf | invalid byte sequence for encoding "MULE_INTERNAL": 0x8b 0x00 +(10 rows) + diff --git a/src/test/regress/expected/opr_sanity.out b/src/test/regress/expected/opr_sanity.out index 254ca06d3dd..23ba60e395f 100644 --- a/src/test/regress/expected/opr_sanity.out +++ b/src/test/regress/expected/opr_sanity.out @@ -1052,13 +1052,14 @@ WHERE p1.conproc = 0 OR SELECT p.oid, p.proname, c.oid, c.conname FROM pg_proc p, pg_conversion c WHERE p.oid = c.conproc AND - (p.prorettype != 'void'::regtype OR p.proretset OR - p.pronargs != 5 OR + (p.prorettype != 'int4'::regtype OR p.proretset OR + p.pronargs != 6 OR p.proargtypes[0] != 'int4'::regtype OR p.proargtypes[1] != 'int4'::regtype OR p.proargtypes[2] != 'cstring'::regtype OR p.proargtypes[3] != 'internal'::regtype OR - p.proargtypes[4] != 'int4'::regtype); + p.proargtypes[4] != 'int4'::regtype OR + p.proargtypes[5] != 'bool'::regtype); oid | proname | oid | conname -----+---------+-----+--------- (0 rows) diff --git a/src/test/regress/input/create_function_1.source b/src/test/regress/input/create_function_1.source index 412e339fcf2..6ba37fe63b6 100644 --- a/src/test/regress/input/create_function_1.source +++ b/src/test/regress/input/create_function_1.source @@ -78,6 +78,10 @@ CREATE FUNCTION test_opclass_options_func(internal) AS '@libdir@/regress@DLSUFFIX@', 'test_opclass_options_func' LANGUAGE C; +CREATE FUNCTION test_enc_conversion(bytea, name, name, bool, validlen OUT int, result OUT bytea) + AS '@libdir@/regress@DLSUFFIX@', 'test_enc_conversion' + LANGUAGE C; + -- Things that shouldn't work: CREATE FUNCTION test1 (int) RETURNS int LANGUAGE SQL diff --git a/src/test/regress/output/create_function_1.source b/src/test/regress/output/create_function_1.source index 4d78fa12289..cb38a039bf4 100644 --- a/src/test/regress/output/create_function_1.source +++ b/src/test/regress/output/create_function_1.source @@ -68,6 +68,9 @@ CREATE FUNCTION test_opclass_options_func(internal) RETURNS void AS '@libdir@/regress@DLSUFFIX@', 'test_opclass_options_func' LANGUAGE C; +CREATE FUNCTION test_enc_conversion(bytea, name, name, bool, validlen OUT int, result OUT bytea) + AS '@libdir@/regress@DLSUFFIX@', 'test_enc_conversion' + LANGUAGE C; -- Things that shouldn't work: CREATE FUNCTION test1 (int) RETURNS int LANGUAGE SQL AS 'SELECT ''not an integer'';'; diff --git a/src/test/regress/regress.c b/src/test/regress/regress.c index 32ab9ed6b53..1990cbb6a13 100644 --- a/src/test/regress/regress.c +++ b/src/test/regress/regress.c @@ -23,12 +23,15 @@ #include "access/htup_details.h" #include "access/transam.h" #include "access/xact.h" +#include "catalog/namespace.h" #include "catalog/pg_operator.h" #include "catalog/pg_type.h" #include "commands/sequence.h" #include "commands/trigger.h" #include "executor/executor.h" #include "executor/spi.h" +#include "funcapi.h" +#include "mb/pg_wchar.h" #include "miscadmin.h" #include "nodes/supportnodes.h" #include "optimizer/optimizer.h" @@ -1060,3 +1063,134 @@ test_opclass_options_func(PG_FUNCTION_ARGS) { PG_RETURN_NULL(); } + +/* + * Call an encoding conversion or verification function. + * + * Arguments: + * string bytea -- string to convert + * src_enc name -- source encoding + * dest_enc name -- destination encoding + * noError bool -- if set, don't ereport() on invalid or untranslatable + * input + * + * Result is a tuple with two attributes: + * int4 -- number of input bytes successfully converted + * bytea -- converted string + */ +PG_FUNCTION_INFO_V1(test_enc_conversion); +Datum +test_enc_conversion(PG_FUNCTION_ARGS) +{ + bytea *string = PG_GETARG_BYTEA_PP(0); + char *src_encoding_name = NameStr(*PG_GETARG_NAME(1)); + int src_encoding = pg_char_to_encoding(src_encoding_name); + char *dest_encoding_name = NameStr(*PG_GETARG_NAME(2)); + int dest_encoding = pg_char_to_encoding(dest_encoding_name); + bool noError = PG_GETARG_BOOL(3); + TupleDesc tupdesc; + char *src; + char *dst; + bytea *retval; + Size srclen; + Size dstsize; + Oid proc; + int convertedbytes; + int dstlen; + Datum values[2]; + bool nulls[2]; + HeapTuple tuple; + + if (src_encoding < 0) + ereport(ERROR, + (errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("invalid source encoding name \"%s\"", + src_encoding_name))); + if (dest_encoding < 0) + ereport(ERROR, + (errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("invalid destination encoding name \"%s\"", + dest_encoding_name))); + + /* Build a tuple descriptor for our result type */ + if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE) + elog(ERROR, "return type must be a row type"); + tupdesc = BlessTupleDesc(tupdesc); + + srclen = VARSIZE_ANY_EXHDR(string); + src = VARDATA_ANY(string); + + if (src_encoding == dest_encoding) + { + /* just check that the source string is valid */ + int oklen; + + oklen = pg_encoding_verifymbstr(src_encoding, src, srclen); + + if (oklen == srclen) + { + convertedbytes = oklen; + retval = string; + } + else if (!noError) + { + report_invalid_encoding(src_encoding, src + oklen, srclen - oklen); + } + else + { + /* + * build bytea data type structure. + */ + Assert(oklen < srclen); + convertedbytes = oklen; + retval = (bytea *) palloc(oklen + VARHDRSZ); + SET_VARSIZE(retval, oklen + VARHDRSZ); + memcpy(VARDATA(retval), src, oklen); + } + } + else + { + proc = FindDefaultConversionProc(src_encoding, dest_encoding); + if (!OidIsValid(proc)) + ereport(ERROR, + (errcode(ERRCODE_UNDEFINED_FUNCTION), + errmsg("default conversion function for encoding \"%s\" to \"%s\" does not exist", + pg_encoding_to_char(src_encoding), + pg_encoding_to_char(dest_encoding)))); + + if (srclen >= (MaxAllocSize / (Size) MAX_CONVERSION_GROWTH)) + ereport(ERROR, + (errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED), + errmsg("out of memory"), + errdetail("String of %d bytes is too long for encoding conversion.", + (int) srclen))); + + dstsize = (Size) srclen * MAX_CONVERSION_GROWTH + 1; + dst = MemoryContextAlloc(CurrentMemoryContext, dstsize); + + /* perform conversion */ + convertedbytes = pg_do_encoding_conversion_buf(proc, + src_encoding, + dest_encoding, + (unsigned char *) src, srclen, + (unsigned char *) dst, dstsize, + noError); + dstlen = strlen(dst); + + /* + * build bytea data type structure. + */ + retval = (bytea *) palloc(dstlen + VARHDRSZ); + SET_VARSIZE(retval, dstlen + VARHDRSZ); + memcpy(VARDATA(retval), dst, dstlen); + + pfree(dst); + } + + MemSet(nulls, 0, sizeof(nulls)); + values[0] = Int32GetDatum(convertedbytes); + values[1] = PointerGetDatum(retval); + tuple = heap_form_tuple(tupdesc, values, nulls); + + PG_RETURN_DATUM(HeapTupleGetDatum(tuple)); +} diff --git a/src/test/regress/sql/conversion.sql b/src/test/regress/sql/conversion.sql index 02cf39f1ce9..ea85f20ed83 100644 --- a/src/test/regress/sql/conversion.sql +++ b/src/test/regress/sql/conversion.sql @@ -34,3 +34,188 @@ DROP CONVERSION mydef; -- RESET SESSION AUTHORIZATION; DROP USER regress_conversion_user; + +-- +-- Test built-in conversion functions. +-- + +-- Helper function to test a conversion. Uses the test_enc_conversion function +-- that was created in the create_function_1 test. +create or replace function test_conv( + input IN bytea, + src_encoding IN text, + dst_encoding IN text, + + result OUT bytea, + errorat OUT bytea, + error OUT text) +language plpgsql as +$$ +declare + validlen int; +begin + -- First try to perform the conversion with noError = false. If that errors out, + -- capture the error message, and try again with noError = true. The second call + -- should succeed and return the position of the error, return that too. + begin + select * into validlen, result from test_enc_conversion(input, src_encoding, dst_encoding, false); + errorat = NULL; + error := NULL; + exception when others then + error := sqlerrm; + select * into validlen, result from test_enc_conversion(input, src_encoding, dst_encoding, true); + errorat = substr(input, validlen + 1); + end; + return; +end; +$$; + + +-- +-- UTF-8 +-- +CREATE TABLE utf8_inputs (inbytes bytea, description text); +insert into utf8_inputs values + ('\x666f6f', 'valid, pure ASCII'), + ('\xc3a4c3b6', 'valid, extra latin chars'), + ('\xd184d0bed0be', 'valid, cyrillic'), + ('\x666f6fe8b1a1', 'valid, kanji/Chinese'), + ('\xe382abe3829a', 'valid, two chars that combine to one in EUC_JIS_2004'), + ('\xe382ab', 'only first half of combined char in EUC_JIS_2004'), + ('\xe382abe382', 'incomplete combination when converted EUC_JIS_2004'), + ('\xecbd94eb81bceba6ac', 'valid, Hangul, Korean'), + ('\x666f6fefa8aa', 'valid, needs mapping function to convert to GB18030'), + ('\x66e8b1ff6f6f', 'invalid byte sequence'), + ('\x66006f', 'invalid, NUL byte'), + ('\x666f6fe8b100', 'invalid, NUL byte'), + ('\x666f6fe8b1', 'incomplete character at end'); + +-- Test UTF-8 verification +select description, (test_conv(inbytes, 'utf8', 'utf8')).* from utf8_inputs; +-- Test conversions from UTF-8 +select description, inbytes, (test_conv(inbytes, 'utf8', 'euc_jis_2004')).* from utf8_inputs; +select description, inbytes, (test_conv(inbytes, 'utf8', 'latin1')).* from utf8_inputs; +select description, inbytes, (test_conv(inbytes, 'utf8', 'latin2')).* from utf8_inputs; +select description, inbytes, (test_conv(inbytes, 'utf8', 'latin5')).* from utf8_inputs; +select description, inbytes, (test_conv(inbytes, 'utf8', 'koi8r')).* from utf8_inputs; +select description, inbytes, (test_conv(inbytes, 'utf8', 'gb18030')).* from utf8_inputs; + +-- +-- EUC_JIS_2004 +-- +CREATE TABLE euc_jis_2004_inputs (inbytes bytea, description text); +insert into euc_jis_2004_inputs values + ('\x666f6f', 'valid, pure ASCII'), + ('\x666f6fbedd', 'valid'), + ('\xa5f7', 'valid, translates to two UTF-8 chars '), + ('\xbeddbe', 'incomplete char '), + ('\x666f6f00bedd', 'invalid, NUL byte'), + ('\x666f6fbe00dd', 'invalid, NUL byte'), + ('\x666f6fbedd00', 'invalid, NUL byte'), + ('\xbe04', 'invalid byte sequence'); + +-- Test EUC_JIS_2004 verification +select description, inbytes, (test_conv(inbytes, 'euc_jis_2004', 'euc_jis_2004')).* from euc_jis_2004_inputs; +-- Test conversions from EUC_JIS_2004 +select description, inbytes, (test_conv(inbytes, 'euc_jis_2004', 'utf8')).* from euc_jis_2004_inputs; + +-- +-- SHIFT-JIS-2004 +-- +CREATE TABLE shiftjis2004_inputs (inbytes bytea, description text); +insert into shiftjis2004_inputs values + ('\x666f6f', 'valid, pure ASCII'), + ('\x666f6f8fdb', 'valid'), + ('\x666f6f81c0', 'valid, no translation to UTF-8'), + ('\x666f6f82f5', 'valid, translates to two UTF-8 chars '), + ('\x666f6f8fdb8f', 'incomplete char '), + ('\x666f6f820a', 'incomplete char, followed by newline '), + ('\x666f6f008fdb', 'invalid, NUL byte'), + ('\x666f6f8f00db', 'invalid, NUL byte'), + ('\x666f6f8fdb00', 'invalid, NUL byte'); + +-- Test SHIFT-JIS-2004 verification +select description, inbytes, (test_conv(inbytes, 'shiftjis2004', 'shiftjis2004')).* from shiftjis2004_inputs; +-- Test conversions from SHIFT-JIS-2004 +select description, inbytes, (test_conv(inbytes, 'shiftjis2004', 'utf8')).* from shiftjis2004_inputs; +select description, inbytes, (test_conv(inbytes, 'shiftjis2004', 'euc_jis_2004')).* from shiftjis2004_inputs; + +-- +-- GB18030 +-- +CREATE TABLE gb18030_inputs (inbytes bytea, description text); +insert into gb18030_inputs values + ('\x666f6f', 'valid, pure ASCII'), + ('\x666f6fcff3', 'valid'), + ('\x666f6f8431a530', 'valid, no translation to UTF-8'), + ('\x666f6f84309c38', 'valid, translates to UTF-8 by mapping function'), + ('\x666f6f84309c', 'incomplete char '), + ('\x666f6f84309c0a', 'incomplete char, followed by newline '), + ('\x666f6f84309c3800', 'invalid, NUL byte'), + ('\x666f6f84309c0038', 'invalid, NUL byte'); + +-- Test GB18030 verification +select description, inbytes, (test_conv(inbytes, 'gb18030', 'gb18030')).* from gb18030_inputs; +-- Test conversions from GB18030 +select description, inbytes, (test_conv(inbytes, 'gb18030', 'utf8')).* from gb18030_inputs; + + +-- +-- ISO-8859-5 +-- +CREATE TABLE iso8859_5_inputs (inbytes bytea, description text); +insert into iso8859_5_inputs values + ('\x666f6f', 'valid, pure ASCII'), + ('\xe4dede', 'valid'), + ('\x00', 'invalid, NUL byte'), + ('\xe400dede', 'invalid, NUL byte'), + ('\xe4dede00', 'invalid, NUL byte'); + +-- Test ISO-8859-5 verification +select description, inbytes, (test_conv(inbytes, 'iso8859-5', 'iso8859-5')).* from iso8859_5_inputs; +-- Test conversions from ISO-8859-5 +select description, inbytes, (test_conv(inbytes, 'iso8859-5', 'utf8')).* from iso8859_5_inputs; +select description, inbytes, (test_conv(inbytes, 'iso8859-5', 'koi8r')).* from iso8859_5_inputs; +select description, inbytes, (test_conv(inbytes, 'iso8859_5', 'mule_internal')).* from iso8859_5_inputs; + +-- +-- Big5 +-- +CREATE TABLE big5_inputs (inbytes bytea, description text); +insert into big5_inputs values + ('\x666f6f', 'valid, pure ASCII'), + ('\x666f6fb648', 'valid'), + ('\x666f6fa27f', 'valid, no translation to UTF-8'), + ('\x666f6fb60048', 'invalid, NUL byte'), + ('\x666f6fb64800', 'invalid, NUL byte'); + +-- Test Big5 verification +select description, inbytes, (test_conv(inbytes, 'big5', 'big5')).* from big5_inputs; +-- Test conversions from Big5 +select description, inbytes, (test_conv(inbytes, 'big5', 'utf8')).* from big5_inputs; +select description, inbytes, (test_conv(inbytes, 'big5', 'mule_internal')).* from big5_inputs; + +-- +-- MULE_INTERNAL +-- +CREATE TABLE mic_inputs (inbytes bytea, description text); +insert into mic_inputs values + ('\x666f6f', 'valid, pure ASCII'), + ('\x8bc68bcf8bcf', 'valid (in KOI8R)'), + ('\x8bc68bcf8b', 'invalid,incomplete char'), + ('\x92bedd', 'valid (in SHIFT_JIS)'), + ('\x92be', 'invalid, incomplete char)'), + ('\x666f6f95a3c1', 'valid (in Big5)'), + ('\x666f6f95a3', 'invalid, incomplete char'), + ('\x9200bedd', 'invalid, NUL byte'), + ('\x92bedd00', 'invalid, NUL byte'), + ('\x8b00c68bcf8bcf', 'invalid, NUL byte'); + +-- Test MULE_INTERNAL verification +select description, inbytes, (test_conv(inbytes, 'mule_internal', 'mule_internal')).* from mic_inputs; +-- Test conversions from MULE_INTERNAL +select description, inbytes, (test_conv(inbytes, 'mule_internal', 'koi8r')).* from mic_inputs; +select description, inbytes, (test_conv(inbytes, 'mule_internal', 'iso8859-5')).* from mic_inputs; +select description, inbytes, (test_conv(inbytes, 'mule_internal', 'sjis')).* from mic_inputs; +select description, inbytes, (test_conv(inbytes, 'mule_internal', 'big5')).* from mic_inputs; +select description, inbytes, (test_conv(inbytes, 'mule_internal', 'euc_jp')).* from mic_inputs; diff --git a/src/test/regress/sql/opr_sanity.sql b/src/test/regress/sql/opr_sanity.sql index bbd3834b634..04691745981 100644 --- a/src/test/regress/sql/opr_sanity.sql +++ b/src/test/regress/sql/opr_sanity.sql @@ -556,13 +556,14 @@ WHERE p1.conproc = 0 OR SELECT p.oid, p.proname, c.oid, c.conname FROM pg_proc p, pg_conversion c WHERE p.oid = c.conproc AND - (p.prorettype != 'void'::regtype OR p.proretset OR - p.pronargs != 5 OR + (p.prorettype != 'int4'::regtype OR p.proretset OR + p.pronargs != 6 OR p.proargtypes[0] != 'int4'::regtype OR p.proargtypes[1] != 'int4'::regtype OR p.proargtypes[2] != 'cstring'::regtype OR p.proargtypes[3] != 'internal'::regtype OR - p.proargtypes[4] != 'int4'::regtype); + p.proargtypes[4] != 'int4'::regtype OR + p.proargtypes[5] != 'bool'::regtype); -- Check for conprocs that don't perform the specific conversion that -- pg_conversion alleges they do, by trying to invoke each conversion -- 2.30.0 --------------1D68F301AB7A25998BA3365F Content-Type: text/x-patch; charset=UTF-8; name="v4-0002-Do-COPY-FROM-encoding-conversion-verification-in-.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename*0="v4-0002-Do-COPY-FROM-encoding-conversion-verification-in-.pa"; filename*1="tch" ^ permalink raw reply [nested|flat] 3+ messages in thread
* Add system column support to the USING clause @ 2024-09-13 08:03 Denis Garsh <[email protected]> 0 siblings, 1 reply; 3+ messages in thread From: Denis Garsh @ 2024-09-13 08:03 UTC (permalink / raw) To: [email protected] <[email protected]> ------==--bound.60954.kxm5fi5dn62wfkjr.vla.yp-c.yandex.net Content-Transfer-Encoding: base64 Content-Type: text/html; charset=utf-8 PGRpdj48ZGl2PjxkaXY+SGVsbG8uIFRoaXMgaXMgbXkgZmlyc3QgcGF0Y2ggdG8gdGhlIHByb2pl Y3QuPC9kaXY+PGRpdj7CoDwvZGl2PjxkaXY+VGhlIHBhdGNoIGFkZHMgc3VwcG9ydCBmb3Igc3lz dGVtIGNvbHVtbnMgaW4gSk9JTiBVU0lORyBjbGF1c2UuIMKgVGhlIHByb2JsZW08L2Rpdj48ZGl2 PmNhbiBiZSBkZW1vbnN0cmF0ZWQgd2l0aCB0aGlzIGNvZGU6PC9kaXY+PGRpdj7CoDwvZGl2Pjxk aXY+YGBgc3FsPC9kaXY+PGRpdj5DUkVBVEUgVEFCTEUgdCDCoChpZCBpbnQpOzwvZGl2PjxkaXY+ Q1JFQVRFIFRBQkxFIHR0IChpZCBpbnQpOzwvZGl2PjxkaXY+wqA8L2Rpdj48ZGl2Pi0tIFdvcmtz OjwvZGl2PjxkaXY+U0VMRUNUICogRlJPTSB0IEpPSU4gdHQgT04gdC54bWluID0gdHQueG1pbjs8 L2Rpdj48ZGl2Pi0tIERvZXNuJ3Qgd29yazo8L2Rpdj48ZGl2PlNFTEVDVCAqIEZST00gdCBKT0lO IHR0IFVTSU5HICh4bWluKTs8L2Rpdj48ZGl2PmBgYDwvZGl2PjxkaXY+wqA8L2Rpdj48ZGl2PlNv bHV0aW9uOjwvZGl2PjxkaXY+MS4gVXNlIHRoZSBzY2FuTlNJdGVtRm9yQ29sdW1uKCkgZnVuY3Rp b24gaW5zdGVhZCBvZiBidWlsZFZhckZyb21OU0NvbHVtbigpIGZvcjwvZGl2PjxkaXY+Y29uc3Ry dWN0aW5nIFZhciBvYmplY3RzLCBhcyBpdCBjb3JyZWN0bHkgaGFuZGxlcyBzeXN0ZW0gY29sdW1u cy48L2Rpdj48ZGl2PjIuIFJlbW92ZSBleHRyYSBjYWxscyB0byBtYXJrVmFyRm9yU2VsZWN0UHJp digpLCBzaW5jZSB0aGlzIGZ1bmN0aW9uIGlzIGFscmVhZHk8L2Rpdj48ZGl2Pmludm9rZWQgaW4g c2Nhbk5TSXRlbUZvckNvbHVtbigpLjwvZGl2PjxkaXY+My4gRm9yIHN5c3RlbSBjb2x1bW5zLCBj b2xsZWN0IHRoZWlyIG5lZ2F0aXZlIGF0dHJpYnV0ZSBudW1iZXJzIGFsb25nIHdpdGg8L2Rpdj48 ZGl2PnVzZXItZGVmaW5lZCBjb2x1bW4gaW5kaWNlcyBpbnRvIGxfY29sbm9zIGFuZCByX2NvbG5v cy48L2Rpdj48ZGl2PjQuIENyZWF0ZSBhIGJ1aWxkVmFyRnJvbVN5c3RlbUF0dHJpYnV0ZSgpIGZ1 bmN0aW9uIGZvciByZWJ1aWxkaW5nIFZhciBvYmplY3RzPC9kaXY+PGRpdj53aXRoIHN5c3RlbSBh dHRyaWJ1dGVzLCBhbmFsb2dvdXMgdG8gYnVpbGRWYXJGcm9tTlNDb2x1bW4oKSwgc2luY2U8L2Rp dj48ZGl2PnNjYW5OU0l0ZW1Gb3JDb2x1bW4oKSBpcyBjb21wbGV4IGFuZCBoYXMgc2lkZSBlZmZl Y3RzLjwvZGl2PjxkaXY+NS4gSW1wbGVtZW50IGEgZmlsbE5TQ29sdW1uUGFyYW1ldGVyc0Zyb21W YXIoKSBmdW5jdGlvbiBmb3IgYnVpbGRpbmcgTlMgY29sdW1uczwvZGl2PjxkaXY+d2l0aCBzeXN0 ZW0gYXR0cmlidXRlcy48L2Rpdj48ZGl2PjxkaXY+Ni4gQWRkIFN5c3RlbUF0dHJpYnV0ZVRvdGFs TnVtYmVyKCkgZnVuY3Rpb24gdG8gaGVhcC5jIHRvIGVuc3VyZSBtZW1vcnkgZm9yPC9kaXY+PGRp dj5yZXNfbnNjb2x1bW5zIGlzIGFsbG9jYXRlZCB3aXRoIHN5c3RlbSBjb2x1bW5zIGluIG1pbmQu PC9kaXY+PC9kaXY+PGRpdj7CoDwvZGl2PjxkaXY+TGluayB0byBQUiBvbiBHaXRIdWI6IGh0dHBz Oi8vZ2l0aHViLmNvbS9oaWxsdHJhY2VyL3Bvc3RncmVzL3B1bGwvMzwvZGl2PjwvZGl2PjwvZGl2 PjxkaXY+LS3CoDwvZGl2PjxkaXY+QmVzdCByZWdhcmRzLCDCoDwvZGl2PjxkaXY+RGVuaXMgR2Fy c2gsIMKgPC9kaXY+PGRpdj5kLmdhcnNoQGFyZW5hZGF0YS5pbzwvZGl2PjxkaXY+wqA8L2Rpdj4= ------==--bound.60954.kxm5fi5dn62wfkjr.vla.yp-c.yandex.net Content-Disposition: attachment; filename="v1-0001-Add-system-column-support-to-the-USING-clause.patch" Content-Transfer-Encoding: base64 Content-Type: text/x-diff; name="v1-0001-Add-system-column-support-to-the-USING-clause.patch" RnJvbSAyOTVmOWVjNTliMjJjYzM3MDdmYjhmMTY4ZTBlMzc1MGU1MTcyNGEwIE1vbiBTZXAgMTcg MDA6MDA6MDAgMjAwMQpGcm9tOiBEZW5pcyBHYXJzaCA8ZC5nYXJzaEBhcmVuYWRhdGEuaW8+CkRh dGU6IFdlZCwgNyBBdWcgMjAyNCAxNTo1NzozMiArMDMwMApTdWJqZWN0OiBbUEFUQ0hdIEFkZCBz eXN0ZW0gY29sdW1uIHN1cHBvcnQgdG8gdGhlIFVTSU5HIGNsYXVzZQoKUHJvYmxlbTogVGhlIHBh cnNlciBjYW4ndCBoYW5kbGUgVVNJTkcgam9pbnMgd2l0aCBzeXN0ZW0gY29sdW1ucywgc3VjaCBh cyB4bWluLgoKU29sdXRpb246CjEuIFVzZSB0aGUgc2Nhbk5TSXRlbUZvckNvbHVtbigpIGZ1bmN0 aW9uIGluc3RlYWQgb2YgYnVpbGRWYXJGcm9tTlNDb2x1bW4oKSBmb3IKY29uc3RydWN0aW5nIFZh ciBvYmplY3RzLCBhcyBpdCBjb3JyZWN0bHkgaGFuZGxlcyBzeXN0ZW0gY29sdW1ucy4KMi4gUmVt b3ZlIGV4dHJhIGNhbGxzIHRvIG1hcmtWYXJGb3JTZWxlY3RQcml2KCksIHNpbmNlIHRoaXMgZnVu Y3Rpb24gaXMgYWxyZWFkeQppbnZva2VkIGluIHNjYW5OU0l0ZW1Gb3JDb2x1bW4oKS4KMy4gRm9y IHN5c3RlbSBjb2x1bW5zLCBjb2xsZWN0IHRoZWlyIG5lZ2F0aXZlIGF0dHJpYnV0ZSBudW1iZXJz IGFsb25nIHdpdGgKdXNlci1kZWZpbmVkIGNvbHVtbiBpbmRpY2VzIGludG8gbF9jb2xub3MgYW5k IHJfY29sbm9zLgo0LiBDcmVhdGUgYSBidWlsZFZhckZyb21TeXN0ZW1BdHRyaWJ1dGUoKSBmdW5j dGlvbiBmb3IgcmVidWlsZGluZyBWYXIgb2JqZWN0cwp3aXRoIHN5c3RlbSBhdHRyaWJ1dGVzLCBh bmFsb2dvdXMgdG8gYnVpbGRWYXJGcm9tTlNDb2x1bW4oKSwgc2luY2UKc2Nhbk5TSXRlbUZvckNv bHVtbigpIGlzIGNvbXBsZXggYW5kIGhhcyBzaWRlIGVmZmVjdHMuCjUuIEltcGxlbWVudCBhIGZp bGxOU0NvbHVtblBhcmFtZXRlcnNGcm9tVmFyKCkgZnVuY3Rpb24gZm9yIGJ1aWxkaW5nIE5TIGNv bHVtbnMKd2l0aCBzeXN0ZW0gYXR0cmlidXRlcy4KNi4gQWRkIFN5c3RlbUF0dHJpYnV0ZVRvdGFs TnVtYmVyKCkgZnVuY3Rpb24gdG8gaGVhcC5jIHRvIGVuc3VyZSBtZW1vcnkgZm9yCnJlc19uc2Nv bHVtbnMgaXMgYWxsb2NhdGVkIHdpdGggc3lzdGVtIGNvbHVtbnMgaW4gbWluZC4KLS0tCiBzcmMv YmFja2VuZC9jYXRhbG9nL2hlYXAuYyAgICAgICAgIHwgICA4ICsrCiBzcmMvYmFja2VuZC9wYXJz ZXIvcGFyc2VfY2xhdXNlLmMgIHwgMTc4ICsrKysrKysrKysrKysrKysrKysrKysrKysrLS0tCiBz cmMvaW5jbHVkZS9jYXRhbG9nL2hlYXAuaCAgICAgICAgIHwgICAyICsKIHNyYy90ZXN0L3JlZ3Jl c3MvZXhwZWN0ZWQvam9pbi5vdXQgfCAxMDYgKysrKysrKysrKysrKysrKysKIHNyYy90ZXN0L3Jl Z3Jlc3Mvc3FsL2pvaW4uc3FsICAgICAgfCAgNTQgKysrKysrKysrCiA1IGZpbGVzIGNoYW5nZWQs IDMzMCBpbnNlcnRpb25zKCspLCAxOCBkZWxldGlvbnMoLSkKCmRpZmYgLS1naXQgYS9zcmMvYmFj a2VuZC9jYXRhbG9nL2hlYXAuYyBiL3NyYy9iYWNrZW5kL2NhdGFsb2cvaGVhcC5jCmluZGV4IDAx YjQzY2M2YTguLmI0NGQwZDAwMDUgMTAwNjQ0Ci0tLSBhL3NyYy9iYWNrZW5kL2NhdGFsb2cvaGVh cC5jCisrKyBiL3NyYy9iYWNrZW5kL2NhdGFsb2cvaGVhcC5jCkBAIC0yNjQsNiArMjY0LDE0IEBA IFN5c3RlbUF0dHJpYnV0ZUJ5TmFtZShjb25zdCBjaGFyICphdHRuYW1lKQogCXJldHVybiBOVUxM OwogfQogCisvKgorICogUmV0dXJucyB0aGUgdG90YWwgbnVtYmVyIG9mIHN5c3RlbSBhdHRyaWJ1 dGVzLgorICovCitzaXplX3QgU3lzdGVtQXR0cmlidXRlVG90YWxOdW1iZXIodm9pZCkKK3sKKwly ZXR1cm4gbGVuZ3Rob2YoU3lzQXR0KTsKK30KKwogCiAvKiAtLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tCiAgKgkJCQlYWFggRU5E IE9GIFVHTFkgSEFSRCBDT0RFRCBCQURORVNTIFhYWApkaWZmIC0tZ2l0IGEvc3JjL2JhY2tlbmQv cGFyc2VyL3BhcnNlX2NsYXVzZS5jIGIvc3JjL2JhY2tlbmQvcGFyc2VyL3BhcnNlX2NsYXVzZS5j CmluZGV4IDgxMTgwMzY0OTUuLjhlOTlkNzgyNzIgMTAwNjQ0Ci0tLSBhL3NyYy9iYWNrZW5kL3Bh cnNlci9wYXJzZV9jbGF1c2UuYworKysgYi9zcmMvYmFja2VuZC9wYXJzZXIvcGFyc2VfY2xhdXNl LmMKQEAgLTI0LDYgKzI0LDcgQEAKICNpbmNsdWRlICJjYXRhbG9nL3BnX2FtcHJvYy5oIgogI2lu Y2x1ZGUgImNhdGFsb2cvcGdfY29uc3RyYWludC5oIgogI2luY2x1ZGUgImNhdGFsb2cvcGdfdHlw ZS5oIgorI2luY2x1ZGUgImNhdGFsb2cvaGVhcC5oIgogI2luY2x1ZGUgImNvbW1hbmRzL2RlZnJl bS5oIgogI2luY2x1ZGUgIm1pc2NhZG1pbi5oIgogI2luY2x1ZGUgIm5vZGVzL21ha2VmdW5jcy5o IgpAQCAtNzYsNiArNzcsMTAgQEAgc3RhdGljIE5vZGUgKnRyYW5zZm9ybUZyb21DbGF1c2VJdGVt KFBhcnNlU3RhdGUgKnBzdGF0ZSwgTm9kZSAqbiwKIAkJCQkJCQkJCSBMaXN0ICoqbmFtZXNwYWNl KTsKIHN0YXRpYyBWYXIgKmJ1aWxkVmFyRnJvbU5TQ29sdW1uKFBhcnNlU3RhdGUgKnBzdGF0ZSwK IAkJCQkJCQkJIFBhcnNlTmFtZXNwYWNlQ29sdW1uICpuc2NvbCk7CitzdGF0aWMgVmFyICpidWls ZFZhckZyb21TeXN0ZW1BdHRyaWJ1dGUoUGFyc2VTdGF0ZSAqcHN0YXRlLAorCQkJCQkJCQkJCWlu dCBydGluZGV4LCBpbnQgYXR0bnVtKTsKK3N0YXRpYyB2b2lkIGZpbGxOU0NvbHVtblBhcmFtZXRl cnNGcm9tVmFyKFBhcnNlTmFtZXNwYWNlQ29sdW1uICpyZXNjb2x1bW4sCisJCQkJCQkJCQkJICBj b25zdCBWYXIgKmNvbHZhcik7CiBzdGF0aWMgTm9kZSAqYnVpbGRNZXJnZWRKb2luVmFyKFBhcnNl U3RhdGUgKnBzdGF0ZSwgSm9pblR5cGUgam9pbnR5cGUsCiAJCQkJCQkJCVZhciAqbF9jb2x2YXIs IFZhciAqcl9jb2x2YXIpOwogc3RhdGljIHZvaWQgbWFya1JlbHNBc051bGxlZEJ5KFBhcnNlU3Rh dGUgKnBzdGF0ZSwgTm9kZSAqbiwgaW50IGppbmRleCk7CkBAIC0yNzMsNyArMjc4LDE2IEBAIGV4 dHJhY3RSZW1haW5pbmdDb2x1bW5zKFBhcnNlU3RhdGUgKnBzdGF0ZSwKIAlwcmV2Y29scyA9IE5V TEw7CiAJZm9yZWFjaChsYywgKnNyY19jb2xub3MpCiAJewotCQlwcmV2Y29scyA9IGJtc19hZGRf bWVtYmVyKHByZXZjb2xzLCBsZmlyc3RfaW50KGxjKSk7CisJCWludCBjb2xfaW5kZXggPSBsZmly c3RfaW50KGxjKTsKKwkJaWYgKGNvbF9pbmRleCA8IDApCisJCXsKKwkJCS8qCisJCQkgKiBBdHRy aWJ1dGUgbnVtYmVyIG9mIHN5c3RlbSBjb2x1bW4uIERvbid0IGNvbGxlY3QgdGhlIG51bWJlcgor CQkJICogYmVjYXVzZSB0aGUgbmFtZSBvZiB0aGlzIGNvbHVtbiBjYW5ub3QgYmUgYW1vbmcgc3Jj X2NvbG5hbWVzLgorCQkJICovCisJCQljb250aW51ZTsKKwkJfQorCQlwcmV2Y29scyA9IGJtc19h ZGRfbWVtYmVyKHByZXZjb2xzLCBjb2xfaW5kZXgpOwogCX0KIAogCWF0dG51bSA9IDA7CkBAIC0z MjcsMTAgKzM0MSw2IEBAIHRyYW5zZm9ybUpvaW5Vc2luZ0NsYXVzZShQYXJzZVN0YXRlICpwc3Rh dGUsCiAJCVZhcgkJICAgKnJ2YXIgPSAoVmFyICopIGxmaXJzdChydmFycyk7CiAJCUFfRXhwcgkg ICAqZTsKIAotCQkvKiBSZXF1aXJlIHJlYWQgYWNjZXNzIHRvIHRoZSBqb2luIHZhcmlhYmxlcyAq LwotCQltYXJrVmFyRm9yU2VsZWN0UHJpdihwc3RhdGUsIGx2YXIpOwotCQltYXJrVmFyRm9yU2Vs ZWN0UHJpdihwc3RhdGUsIHJ2YXIpOwotCiAJCS8qIE5vdyBjcmVhdGUgdGhlIGx2YXIgPSBydmFy IGpvaW4gY29uZGl0aW9uICovCiAJCWUgPSBtYWtlU2ltcGxlQV9FeHByKEFFWFBSX09QLCAiPSIs CiAJCQkJCQkJIChOb2RlICopIGNvcHlPYmplY3QobHZhciksIChOb2RlICopIGNvcHlPYmplY3Qo cnZhciksCkBAIC0xMjkxLDcgKzEzMDEsOSBAQCB0cmFuc2Zvcm1Gcm9tQ2xhdXNlSXRlbShQYXJz ZVN0YXRlICpwc3RhdGUsIE5vZGUgKm4sCiAKIAkJLyogdGhpcyBtYXkgYmUgbGFyZ2VyIHRoYW4g bmVlZGVkLCBidXQgaXQncyBub3Qgd29ydGggYmVpbmcgZXhhY3QgKi8KIAkJcmVzX25zY29sdW1u cyA9IChQYXJzZU5hbWVzcGFjZUNvbHVtbiAqKQotCQkJcGFsbG9jMCgobGlzdF9sZW5ndGgobF9j b2xuYW1lcykgKyBsaXN0X2xlbmd0aChyX2NvbG5hbWVzKSkgKgorCQkJcGFsbG9jMCgobGlzdF9s ZW5ndGgobF9jb2xuYW1lcykgKworCQkJCQkgbGlzdF9sZW5ndGgocl9jb2xuYW1lcykgKworCQkJ CQkgU3lzdGVtQXR0cmlidXRlVG90YWxOdW1iZXIoKSkgKgogCQkJCQlzaXplb2YoUGFyc2VOYW1l c3BhY2VDb2x1bW4pKTsKIAkJcmVzX2NvbGluZGV4ID0gMDsKIApAQCAtMTMzMiw2ICsxMzQ0LDE0 IEBAIHRyYW5zZm9ybUZyb21DbGF1c2VJdGVtKFBhcnNlU3RhdGUgKnBzdGF0ZSwgTm9kZSAqbiwK IAkJCQkJCQkJCQl1X2NvbG5hbWUpKSk7CiAJCQkJfQogCisJCQkJLyoKKwkJCQkgKiBJdCBpcyBp bXBvcnRhbnQgdG8gdW5kZXJzdGFuZCB0aGF0IGFtb25nIHRoZSBsX2NvbG5hbWVzIG9yCisJCQkJ ICogcl9jb2xuYW1lcyB0aGVyZSBtYXkgYmUgYSBzeXN0ZW0gY29sdW1uIG5hbWUuIEEgc3lzdGVt CisJCQkJICogY29sdW1uIG5hbWUgbWF5IGFwcGVhciBpZiBpdCB3YXMgYWRkZWQgaW4gdGhlIHBy ZXZpb3VzIHN0ZXAuCisJCQkJICogVGhpcyBpcyBiZWNhdXNlIGEgcXVlcnkgY2FuIGNvbnRhaW4g bXVsdGlwbGUgVVNJTkcgam9pbnMsCisJCQkJICogd2hlcmUgZWFjaCBqb2luIHVzZXMgdGhlIHJl c3VsdCBvZiB0aGUgcHJldmlvdXMgb25lLgorCQkJCSAqLworCiAJCQkJLyogRmluZCBpdCBpbiBs ZWZ0IGlucHV0ICovCiAJCQkJbmR4ID0gMDsKIAkJCQlmb3JlYWNoKGNvbCwgbF9jb2xuYW1lcykK QEAgLTEzNDksMTIgKzEzNjksMjUgQEAgdHJhbnNmb3JtRnJvbUNsYXVzZUl0ZW0oUGFyc2VTdGF0 ZSAqcHN0YXRlLCBOb2RlICpuLAogCQkJCQl9CiAJCQkJCW5keCsrOwogCQkJCX0KLQkJCQlpZiAo bF9pbmRleCA8IDApCisKKwkJCQlsX2NvbHZhciA9IChWYXIgKikgc2Nhbk5TSXRlbUZvckNvbHVt bihwc3RhdGUsCisJCQkJCQkJCQkJCQkJICAgbF9uc2l0ZW0sCisJCQkJCQkJCQkJCQkJICAgMCwg dV9jb2xuYW1lLCAtMSk7CisJCQkJaWYgKGxfY29sdmFyID09IE5VTEwpCiAJCQkJCWVyZXBvcnQo RVJST1IsCiAJCQkJCQkJKGVycmNvZGUoRVJSQ09ERV9VTkRFRklORURfQ09MVU1OKSwKIAkJCQkJ CQkgZXJybXNnKCJjb2x1bW4gXCIlc1wiIHNwZWNpZmllZCBpbiBVU0lORyBjbGF1c2UgZG9lcyBu b3QgZXhpc3QgaW4gbGVmdCB0YWJsZSIsCiAJCQkJCQkJCQl1X2NvbG5hbWUpKSk7Ci0JCQkJbF9j b2xub3MgPSBsYXBwZW5kX2ludChsX2NvbG5vcywgbF9pbmRleCArIDEpOworCQkJCWlmIChsX2lu ZGV4ID49IDApCisJCQkJeworCQkJCQkvKiBVc2VyLWRlZmluZWQgY29sdW1uICovCisJCQkJCWxf Y29sbm9zID0gbGFwcGVuZF9pbnQobF9jb2xub3MsIGxfaW5kZXggKyAxKTsKKwkJCQl9CisJCQkJ ZWxzZQorCQkJCXsKKwkJCQkJLyogU3lzdGVtIGNvbHVtbiAqLworCQkJCQlsX2NvbG5vcyA9IGxh cHBlbmRfaW50KGxfY29sbm9zLCBsX2NvbHZhci0+dmFyYXR0bm8pOworCQkJCX0KIAogCQkJCS8q IEZpbmQgaXQgaW4gcmlnaHQgaW5wdXQgKi8KIAkJCQluZHggPSAwOwpAQCAtMTM3MywxNyArMTQw NiwzMCBAQCB0cmFuc2Zvcm1Gcm9tQ2xhdXNlSXRlbShQYXJzZVN0YXRlICpwc3RhdGUsIE5vZGUg Km4sCiAJCQkJCX0KIAkJCQkJbmR4Kys7CiAJCQkJfQotCQkJCWlmIChyX2luZGV4IDwgMCkKKwor CQkJCXJfY29sdmFyID0gKFZhciAqKSBzY2FuTlNJdGVtRm9yQ29sdW1uKHBzdGF0ZSwKKwkJCQkJ CQkJCQkJCQkgICByX25zaXRlbSwKKwkJCQkJCQkJCQkJCQkgICAwLCB1X2NvbG5hbWUsIC0xKTsK KworCQkJCWlmIChyX2NvbHZhciA9PSBOVUxMKQogCQkJCQllcmVwb3J0KEVSUk9SLAogCQkJCQkJ CShlcnJjb2RlKEVSUkNPREVfVU5ERUZJTkVEX0NPTFVNTiksCiAJCQkJCQkJIGVycm1zZygiY29s dW1uIFwiJXNcIiBzcGVjaWZpZWQgaW4gVVNJTkcgY2xhdXNlIGRvZXMgbm90IGV4aXN0IGluIHJp Z2h0IHRhYmxlIiwKIAkJCQkJCQkJCXVfY29sbmFtZSkpKTsKLQkJCQlyX2NvbG5vcyA9IGxhcHBl bmRfaW50KHJfY29sbm9zLCByX2luZGV4ICsgMSk7CisKKwkJCQlpZiAocl9pbmRleCA+PSAwKQor CQkJCXsKKwkJCQkJLyogVXNlci1kZWZpbmVkIGNvbHVtbiAqLworCQkJCQlyX2NvbG5vcyA9IGxh cHBlbmRfaW50KHJfY29sbm9zLCByX2luZGV4ICsgMSk7CisJCQkJfQorCQkJCWVsc2UKKwkJCQl7 CisJCQkJCS8qIFN5c3RlbSBjb2x1bW4gKi8KKwkJCQkJcl9jb2xub3MgPSBsYXBwZW5kX2ludChy X2NvbG5vcywgcl9jb2x2YXItPnZhcmF0dG5vKTsKKwkJCQl9CiAKIAkJCQkvKiBCdWlsZCBWYXJz IHRvIHVzZSBpbiB0aGUgZ2VuZXJhdGVkIEpPSU4gT04gY2xhdXNlICovCi0JCQkJbF9jb2x2YXIg PSBidWlsZFZhckZyb21OU0NvbHVtbihwc3RhdGUsIGxfbnNjb2x1bW5zICsgbF9pbmRleCk7CiAJ CQkJbF91c2luZ3ZhcnMgPSBsYXBwZW5kKGxfdXNpbmd2YXJzLCBsX2NvbHZhcik7Ci0JCQkJcl9j b2x2YXIgPSBidWlsZFZhckZyb21OU0NvbHVtbihwc3RhdGUsIHJfbnNjb2x1bW5zICsgcl9pbmRl eCk7CiAJCQkJcl91c2luZ3ZhcnMgPSBsYXBwZW5kKHJfdXNpbmd2YXJzLCByX2NvbHZhcik7CiAK IAkJCQkvKgpAQCAtMTQ2MSw4ICsxNTA3LDggQEAgdHJhbnNmb3JtRnJvbUNsYXVzZUl0ZW0oUGFy c2VTdGF0ZSAqcHN0YXRlLCBOb2RlICpuLAogCQkJLyogU2NhbiB0aGUgY29sbm9zIGxpc3RzIHRv IHJlY292ZXIgaW5mbyBmcm9tIHRoZSBwcmV2aW91cyBsb29wICovCiAJCQlmb3Jib3RoKGxjMSwg bF9jb2xub3MsIGxjMiwgcl9jb2xub3MpCiAJCQl7Ci0JCQkJaW50CQkJbF9pbmRleCA9IGxmaXJz dF9pbnQobGMxKSAtIDE7Ci0JCQkJaW50CQkJcl9pbmRleCA9IGxmaXJzdF9pbnQobGMyKSAtIDE7 CisJCQkJaW50CQkJbF9pbmRleCA9IGxmaXJzdF9pbnQobGMxKTsKKwkJCQlpbnQJCQlyX2luZGV4 ID0gbGZpcnN0X2ludChsYzIpOwogCQkJCVZhcgkJICAgKmxfY29sdmFyLAogCQkJCQkJICAgKnJf Y29sdmFyOwogCQkJCU5vZGUJICAgKnVfY29sdmFyOwpAQCAtMTQ3Miw4ICsxNTE4LDMzIEBAIHRy YW5zZm9ybUZyb21DbGF1c2VJdGVtKFBhcnNlU3RhdGUgKnBzdGF0ZSwgTm9kZSAqbiwKIAkJCQkg KiBOb3RlIHdlIHJlLWJ1aWxkIHRoZXNlIFZhcnM6IHRoZXkgbWlnaHQgaGF2ZSBkaWZmZXJlbnQK IAkJCQkgKiB2YXJudWxsaW5ncmVscyB0aGFuIHRoZSBvbmVzIG1hZGUgaW4gdGhlIHByZXZpb3Vz IGxvb3AuCiAJCQkJICovCi0JCQkJbF9jb2x2YXIgPSBidWlsZFZhckZyb21OU0NvbHVtbihwc3Rh dGUsIGxfbnNjb2x1bW5zICsgbF9pbmRleCk7Ci0JCQkJcl9jb2x2YXIgPSBidWlsZFZhckZyb21O U0NvbHVtbihwc3RhdGUsIHJfbnNjb2x1bW5zICsgcl9pbmRleCk7CisJCQkJaWYgKGxfaW5kZXgg PiAwKQorCQkJCXsKKwkJCQkJLyogVXNlci1kZWZpbmVkIGNvbHVtbiAqLworCQkJCQlsX2NvbHZh ciA9IGJ1aWxkVmFyRnJvbU5TQ29sdW1uKHBzdGF0ZSwKKwkJCQkJCQkJCQkJCQlsX25zY29sdW1u cyArIGxfaW5kZXggLSAxKTsKKwkJCQl9CisJCQkJZWxzZQorCQkJCXsKKwkJCQkJLyogU3lzdGVt IGNvbHVtbiwgc28gbF9pbmRleCBpcyBhdHRudW0uICovCisJCQkJCWxfY29sdmFyID0gYnVpbGRW YXJGcm9tU3lzdGVtQXR0cmlidXRlKHBzdGF0ZSwKKwkJCQkJCQkJCQkJCQkJICAgbF9uc2l0ZW0t PnBfcnRpbmRleCwKKwkJCQkJCQkJCQkJCQkJICAgbF9pbmRleCk7CisJCQkJfQorCisJCQkJaWYg KHJfaW5kZXggPiAwKQorCQkJCXsKKwkJCQkJLyogVXNlci1kZWZpbmVkIGNvbHVtbiAqLworCQkJ CQlyX2NvbHZhciA9IGJ1aWxkVmFyRnJvbU5TQ29sdW1uKHBzdGF0ZSwKKwkJCQkJCQkJCQkJCQly X25zY29sdW1ucyArIHJfaW5kZXggLSAxKTsKKwkJCQl9CisJCQkJZWxzZQorCQkJCXsKKwkJCQkJ LyogU3lzdGVtIGNvbHVtbiwgc28gcl9pbmRleCBpcyBhdHRudW0uICovCisJCQkJCXJfY29sdmFy ID0gYnVpbGRWYXJGcm9tU3lzdGVtQXR0cmlidXRlKHBzdGF0ZSwKKwkJCQkJCQkJCQkJCQkJICAg cl9uc2l0ZW0tPnBfcnRpbmRleCwKKwkJCQkJCQkJCQkJCQkJICAgcl9pbmRleCk7CisJCQkJfQog CiAJCQkJLyogQ29uc3RydWN0IHRoZSBqb2luIGFsaWFzIFZhciBmb3IgdGhpcyBjb2x1bW4gKi8K IAkJCQl1X2NvbHZhciA9IGJ1aWxkTWVyZ2VkSm9pblZhcihwc3RhdGUsCkBAIC0xNDg4LDEyICsx NTU5LDMyIEBAIHRyYW5zZm9ybUZyb21DbGF1c2VJdGVtKFBhcnNlU3RhdGUgKnBzdGF0ZSwgTm9k ZSAqbiwKIAkJCQlpZiAodV9jb2x2YXIgPT0gKE5vZGUgKikgbF9jb2x2YXIpCiAJCQkJewogCQkJ CQkvKiBNZXJnZWQgY29sdW1uIGlzIGVxdWl2YWxlbnQgdG8gbGVmdCBpbnB1dCAqLwotCQkJCQkq cmVzX25zY29sdW1uID0gbF9uc2NvbHVtbnNbbF9pbmRleF07CisJCQkJCWlmIChsX2luZGV4ID4g MCkKKwkJCQkJeworCQkJCQkJLyogVXNlci1kZWZpbmVkIGNvbHVtbiwgc28ganVzdCB0YWtlIGl0 LiAqLworCQkJCQkJKnJlc19uc2NvbHVtbiA9IGxfbnNjb2x1bW5zW2xfaW5kZXggLSAxXTsKKwkJ CQkJfQorCQkJCQllbHNlCisJCQkJCXsKKwkJCQkJCS8qIFN5c3RlbSBjb2x1bW4sIHNvIGNvbnN0 cnVjdCBuZXcgY29sdW1uLiAqLworCQkJCQkJZmlsbE5TQ29sdW1uUGFyYW1ldGVyc0Zyb21WYXIo cmVzX25zY29sdW1uLAorCQkJCQkJCQkJCQkJCSAgbF9jb2x2YXIpOworCQkJCQl9CiAJCQkJfQog CQkJCWVsc2UgaWYgKHVfY29sdmFyID09IChOb2RlICopIHJfY29sdmFyKQogCQkJCXsKIAkJCQkJ LyogTWVyZ2VkIGNvbHVtbiBpcyBlcXVpdmFsZW50IHRvIHJpZ2h0IGlucHV0ICovCi0JCQkJCSpy ZXNfbnNjb2x1bW4gPSByX25zY29sdW1uc1tyX2luZGV4XTsKKwkJCQkJaWYgKHJfaW5kZXggPiAw KQorCQkJCQl7CisJCQkJCQkvKiBVc2VyLWRlZmluZWQgY29sdW1uLCBzbyBqdXN0IHRha2UgaXQu ICovCisJCQkJCQkqcmVzX25zY29sdW1uID0gcl9uc2NvbHVtbnNbcl9pbmRleCAtIDFdOworCQkJ CQl9CisJCQkJCWVsc2UKKwkJCQkJeworCQkJCQkJLyogU3lzdGVtIGNvbHVtbiwgc28gY29uc3Ry dWN0IG5ldyBjb2x1bW4uICovCisJCQkJCQlmaWxsTlNDb2x1bW5QYXJhbWV0ZXJzRnJvbVZhcihy ZXNfbnNjb2x1bW4sCisJCQkJCQkJCQkJCQkJICByX2NvbHZhcik7CisJCQkJCX0KIAkJCQl9CiAJ CQkJZWxzZQogCQkJCXsKQEAgLTE2NTgsNiArMTc0OSw1NyBAQCBidWlsZFZhckZyb21OU0NvbHVt bihQYXJzZVN0YXRlICpwc3RhdGUsIFBhcnNlTmFtZXNwYWNlQ29sdW1uICpuc2NvbCkKIAlyZXR1 cm4gdmFyOwogfQogCisvKgorICogYnVpbGRWYXJGcm9tU3lzdGVtQXR0cmlidXRlIC0KKyAqCSAg YnVpbGQgYSBWYXIgbm9kZSB1c2luZyBzeXN0ZW0gYXR0cmlidXRlLCBlLmcuICJ4bWluIi4KKyAq CisgKiBydGluZGV4IGlzIHRoZSByZWxhdGlvbidzIGluZGV4IGluIHRoZSByYW5nZXRhYmxlLgor ICogYXR0bnVtIGlzIHRoZSBudW1iZXIgb2Ygc3lzdGVtIGF0dHJpYnV0ZS4KKyAqIFJldHVybnMg TlVMTCBpZiB0aGVyZSBpcyBubyBzdWNoIHN5c3RlbSBhdHRyaWJ1dGUuCisgKi8KK3N0YXRpYyBW YXIgKgorYnVpbGRWYXJGcm9tU3lzdGVtQXR0cmlidXRlKFBhcnNlU3RhdGUgKnBzdGF0ZSwgaW50 IHJ0aW5kZXgsIGludCBhdHRudW0pCit7CisJVmFyCQkgICAqdmFyOworCWNvbnN0IEZvcm1EYXRh X3BnX2F0dHJpYnV0ZSAqc3lzYXR0OworCisJc3lzYXR0ID0gU3lzdGVtQXR0cmlidXRlRGVmaW5p dGlvbihhdHRudW0pOworCisJaWYgKHN5c2F0dCA9PSBOVUxMKQorCQlyZXR1cm4gTlVMTDsKKwor CXZhciA9IG1ha2VWYXIocnRpbmRleCwKKwkJCQkgIGF0dG51bSwKKwkJCQkgIHN5c2F0dC0+YXR0 dHlwaWQsCisJCQkJICBzeXNhdHQtPmF0dHR5cG1vZCwKKwkJCQkgIHN5c2F0dC0+YXR0Y29sbGF0 aW9uLAorCQkJCSAgMCk7CisKKwkvKiB1cGRhdGUgdmFybnVsbGluZ3JlbHMgKi8KKwltYXJrTnVs bGFibGVJZk5lZWRlZChwc3RhdGUsIHZhcik7CisJcmV0dXJuIHZhcjsKK30KKworLyoKKyAqIGZp bGxOU0NvbHVtblBhcmFtZXRlcnNGcm9tVmFyIC0KKyAqCSAgZmlsbCBQYXJzZU5hbWVzcGFjZUNv bHVtbiBkYXRhIHVzaW5nIFZhciBkYXRhLgorICoKKyAqIHJlc2NvbHVtbiBpcyBwb2ludGVyIHRv IHRoZSBzdHJ1Y3R1cmUgdG8gYmUgZmlsbGVkLgorICogY29sdmFyIGlzIHRoZSBWYXIgd2hlcmUg dG8gZ2V0IGRhdGEgZnJvbS4KKyAqLworc3RhdGljIHZvaWQKK2ZpbGxOU0NvbHVtblBhcmFtZXRl cnNGcm9tVmFyKFBhcnNlTmFtZXNwYWNlQ29sdW1uICpyZXNjb2x1bW4sCisJCQkJCQkJICBjb25z dCBWYXIgKmNvbHZhcikKK3sKKwlyZXNjb2x1bW4tPnBfdmFybm8gPSBjb2x2YXItPnZhcm5vOwor CXJlc2NvbHVtbi0+cF92YXJhdHRubyA9IGNvbHZhci0+dmFyYXR0bm87CisJcmVzY29sdW1uLT5w X3ZhcnR5cGUgPSBjb2x2YXItPnZhcnR5cGU7CisJcmVzY29sdW1uLT5wX3ZhcnR5cG1vZCA9IGNv bHZhci0+dmFydHlwbW9kOworCXJlc2NvbHVtbi0+cF92YXJjb2xsaWQgPSBjb2x2YXItPnZhcmNv bGxpZDsKKwlyZXNjb2x1bW4tPnBfdmFybm9zeW4gPSBjb2x2YXItPnZhcm5vc3luOworCXJlc2Nv bHVtbi0+cF92YXJhdHRub3N5biA9IGNvbHZhci0+dmFyYXR0bm9zeW47Cit9CisKIC8qCiAgKiBi dWlsZE1lcmdlZEpvaW5WYXIgLQogICoJICBnZW5lcmF0ZSBhIHN1aXRhYmxlIHJlcGxhY2VtZW50 IGV4cHJlc3Npb24gZm9yIGEgbWVyZ2VkIGpvaW4gY29sdW1uCmRpZmYgLS1naXQgYS9zcmMvaW5j bHVkZS9jYXRhbG9nL2hlYXAuaCBiL3NyYy9pbmNsdWRlL2NhdGFsb2cvaGVhcC5oCmluZGV4IGM1 MTI4MjRjZDEuLjk2MGRiMTUyYjUgMTAwNjQ0Ci0tLSBhL3NyYy9pbmNsdWRlL2NhdGFsb2cvaGVh cC5oCisrKyBiL3NyYy9pbmNsdWRlL2NhdGFsb2cvaGVhcC5oCkBAIC0xMzYsNiArMTM2LDggQEAg ZXh0ZXJuIGNvbnN0IEZvcm1EYXRhX3BnX2F0dHJpYnV0ZSAqU3lzdGVtQXR0cmlidXRlRGVmaW5p dGlvbihBdHRyTnVtYmVyIGF0dG5vKTsKIAogZXh0ZXJuIGNvbnN0IEZvcm1EYXRhX3BnX2F0dHJp YnV0ZSAqU3lzdGVtQXR0cmlidXRlQnlOYW1lKGNvbnN0IGNoYXIgKmF0dG5hbWUpOwogCitleHRl cm4gc2l6ZV90IFN5c3RlbUF0dHJpYnV0ZVRvdGFsTnVtYmVyKHZvaWQpOworCiBleHRlcm4gdm9p ZCBDaGVja0F0dHJpYnV0ZU5hbWVzVHlwZXMoVHVwbGVEZXNjIHR1cGRlc2MsIGNoYXIgcmVsa2lu ZCwKIAkJCQkJCQkJCSBpbnQgZmxhZ3MpOwogCmRpZmYgLS1naXQgYS9zcmMvdGVzdC9yZWdyZXNz L2V4cGVjdGVkL2pvaW4ub3V0IGIvc3JjL3Rlc3QvcmVncmVzcy9leHBlY3RlZC9qb2luLm91dApp bmRleCAzMWZiN2QxNDJlLi4xZWE5YTFhYjcxIDEwMDY0NAotLS0gYS9zcmMvdGVzdC9yZWdyZXNz L2V4cGVjdGVkL2pvaW4ub3V0CisrKyBiL3NyYy90ZXN0L3JlZ3Jlc3MvZXhwZWN0ZWQvam9pbi5v dXQKQEAgLTgwMTcsMyArODAxNywxMDkgQEAgU0VMRUNUIHQxLmEgRlJPTSBza2lwX2ZldGNoIHQx IExFRlQgSk9JTiBza2lwX2ZldGNoIHQyIE9OIHQyLmEgPSAxIFdIRVJFIHQyLmEgSVMKIAogUkVT RVQgZW5hYmxlX2luZGV4b25seXNjYW47CiBSRVNFVCBlbmFibGVfc2Vxc2NhbjsKKy0tIFRlc3Qg VVNJTkcgd2l0aCBzeXN0ZW0gY29sdW1ucworLS0gR2VuZXJhdGUgMiB0YWJsZXMgd2l0aCAzIHJv d3MgZWFjaC4KKy0tIHhtaW4gKHRyYW5zYWN0aW9uIG51bWJlcikgaXMgYW4gaW50ZWdlciB2YWx1 ZSByZXByZXNlbnRlZCBhcyBUMSwgVDIsIGV0Yy4KKy0tIFRoZSB4bWluIHZhbHVlcyBvZiB0aGUg Zmlyc3QgYW5kIHNlY29uZCByb3dzIGFyZSBlcXVhbCB0byBlYWNoIG90aGVyLAorLS0gcmVzcGVj dGl2ZWx5LiBUaGUgdmFsdWVzIFQzIGFuZCBUNCBvZiB0aGUgdGhpcmQgcm93cyBhcmUgbm90IGVx dWFsLgorLS0gSjE6IGlkMSB8IHhtaW4gICAgICBKMjogaWQyIHwgeG1pbgorLS0gICAgIC0tLS0r LS0tLS0tLSAgICAgICAtLS0tLSstLS0tLS0tCistLSAgICAgICAxIHwgVDEgICAgICAgICAgICAx MDEgfCBUMQorLS0gICAgICAgMiB8IFQyICAgICAgICAgICAgMTAyIHwgVDIKKy0tICAgICAgIDMg fCBUMyAgICAgICAgICAgIDEwMyB8IFQ0CitDUkVBVEUgVEFCTEUgajEgKGlkMSBJTlQpOworQ1JF QVRFIFRBQkxFIGoyIChpZDIgSU5UKTsKK0JFR0lOOworICBJTlNFUlQgSU5UTyBqMSAoaWQxKSBW QUxVRVMgKDEpOworICBJTlNFUlQgSU5UTyBqMiAoaWQyKSBWQUxVRVMgKDEwMSk7CitDT01NSVQ7 CitCRUdJTjsKKyAgSU5TRVJUIElOVE8gajEgKGlkMSkgVkFMVUVTICgyKTsKKyAgSU5TRVJUIElO VE8gajIgKGlkMikgVkFMVUVTICgxMDIpOworQ09NTUlUOworQkVHSU47CisgIElOU0VSVCBJTlRP IGoxIChpZDEpIFZBTFVFUyAoMyk7CitDT01NSVQ7CitCRUdJTjsKKyAgSU5TRVJUIElOVE8gajIg KGlkMikgVkFMVUVTICgxMDMpOworQ09NTUlUOworLS0gVGhlIHhtaW4gdmFsdWVzIGFyZSBub3Qg b3V0cHV0LCBhcyB0aGV5IHdpbGwgZGlmZmVyIHdpdGggZWFjaCBydW4uCistLSBTbywgb25seSB0 aGUgY29sdW1uIGhlYWRlcnMgYW5kIHRoZSB2YWx1ZXMgb2YgaWQxIGFuZCBpZDIgYXJlIGRpc3Bs YXllZC4KK1NFTEVDVCAqIEZST00gajEgSk9JTiBqMiBVU0lORyAoeG1pbikgTElNSVQgMDsKKyB4 bWluIHwgaWQxIHwgaWQyIAorLS0tLS0tKy0tLS0tKy0tLS0tCisoMCByb3dzKQorCitTRUxFQ1Qg ajEuaWQxLCBqMi5pZDIgRlJPTSBqMSBKT0lOIGoyIFVTSU5HICh4bWluKTsKKyBpZDEgfCBpZDIg CistLS0tLSstLS0tLQorICAgMSB8IDEwMQorICAgMiB8IDEwMgorKDIgcm93cykKKworU0VMRUNU ICogRlJPTSBqMSBMRUZUIEpPSU4gajIgVVNJTkcgKHhtaW4pIExJTUlUIDA7CisgeG1pbiB8IGlk MSB8IGlkMiAKKy0tLS0tLSstLS0tLSstLS0tLQorKDAgcm93cykKKworU0VMRUNUIGoxLmlkMSwg ajIuaWQyIEZST00gajEgTEVGVCBKT0lOIGoyIFVTSU5HICh4bWluKTsKKyBpZDEgfCBpZDIgCist LS0tLSstLS0tLQorICAgMSB8IDEwMQorICAgMiB8IDEwMgorICAgMyB8ICAgIAorKDMgcm93cykK KworU0VMRUNUICogRlJPTSBqMSBSSUdIVCBKT0lOIGoyIFVTSU5HICh4bWluKSBMSU1JVCAwOwor IHhtaW4gfCBpZDEgfCBpZDIgCistLS0tLS0rLS0tLS0rLS0tLS0KKygwIHJvd3MpCisKK1NFTEVD VCBqMS5pZDEsIGoyLmlkMiBGUk9NIGoxIFJJR0hUIEpPSU4gajIgVVNJTkcgKHhtaW4pOworIGlk MSB8IGlkMiAKKy0tLS0tKy0tLS0tCisgICAxIHwgMTAxCisgICAyIHwgMTAyCisgICAgIHwgMTAz CisoMyByb3dzKQorCitTRUxFQ1QgKiBGUk9NIGoxIEZVTEwgSk9JTiBqMiBVU0lORyAoeG1pbikg TElNSVQgMDsKKyB4bWluIHwgaWQxIHwgaWQyIAorLS0tLS0tKy0tLS0tKy0tLS0tCisoMCByb3dz KQorCitTRUxFQ1QgajEuaWQxLCBqMi5pZDIgRlJPTSBqMSBGVUxMIEpPSU4gajIgVVNJTkcgKHht aW4pOworIGlkMSB8IGlkMiAKKy0tLS0tKy0tLS0tCisgICAxIHwgMTAxCisgICAyIHwgMTAyCisg ICAzIHwgICAgCisgICAgIHwgMTAzCisoNCByb3dzKQorCistLSBUZXN0IGlmIFVTSU5HIGNhbiBh ZGQgYWxsIHN5c3RlbSBjb2x1bW5zIGF0IG9uY2UKK1NFTEVDVCAqCisgIEZST00gajEgSk9JTiBq MiBVU0lORyAodGFibGVvaWQsIHhtaW4sIGNtaW4sIHhtYXgsIGNtYXgsIGN0aWQpOworIHRhYmxl b2lkIHwgeG1pbiB8IGNtaW4gfCB4bWF4IHwgY21heCB8IGN0aWQgfCBpZDEgfCBpZDIgCistLS0t LS0tLS0tKy0tLS0tLSstLS0tLS0rLS0tLS0tKy0tLS0tLSstLS0tLS0rLS0tLS0rLS0tLS0KKygw IHJvd3MpCisKKy0tIFRlc3QgVVNJTkcgZXhjZXB0aW9ucworU0VMRUNUICogRlJPTSBqMSBKT0lO IGoyIFVTSU5HIChpZDIpOworRVJST1I6ICBjb2x1bW4gImlkMiIgc3BlY2lmaWVkIGluIFVTSU5H IGNsYXVzZSBkb2VzIG5vdCBleGlzdCBpbiBsZWZ0IHRhYmxlCitTRUxFQ1QgKiBGUk9NIGoxIEpP SU4gajIgVVNJTkcgKGlkMSk7CitFUlJPUjogIGNvbHVtbiAiaWQxIiBzcGVjaWZpZWQgaW4gVVNJ TkcgY2xhdXNlIGRvZXMgbm90IGV4aXN0IGluIHJpZ2h0IHRhYmxlCitXSVRIIGoxX2R1YmJlZCBB UyAoU0VMRUNUIGlkMSwgaWQxIEZST00gajEpCisgIFNFTEVDVCAqIEZST00gajFfZHViYmVkIEpP SU4gajEgVVNJTkcgKGlkMSk7CitFUlJPUjogIGNvbW1vbiBjb2x1bW4gbmFtZSAiaWQxIiBhcHBl YXJzIG1vcmUgdGhhbiBvbmNlIGluIGxlZnQgdGFibGUKK1dJVEggajFfZHViYmVkIEFTIChTRUxF Q1QgaWQxLCBpZDEgRlJPTSBqMSkKKyAgU0VMRUNUICogRlJPTSBqMSBKT0lOIGoxX2R1YmJlZCBV U0lORyAoaWQxKTsKK0VSUk9SOiAgY29tbW9uIGNvbHVtbiBuYW1lICJpZDEiIGFwcGVhcnMgbW9y ZSB0aGFuIG9uY2UgaW4gcmlnaHQgdGFibGUKKy0tIFRlc3QgbXVsdGlwbGUgVVNJTkcgam9pbnMs IGVhY2ggYmFzZWQgb24gdGhlIHByaW9yIHJlc3VsdC4KK0NSRUFURSBUQUJMRSBqMyAoaWQzIElO VCk7CitTRUxFQ1QgKiBGUk9NIGoxIEpPSU4gajIgVVNJTkcgKHhtaW4pIEpPSU4gajMgVVNJTkcg KHhtaW4pIExJTUlUIDA7CisgeG1pbiB8IGlkMSB8IGlkMiB8IGlkMyAKKy0tLS0tLSstLS0tLSst LS0tLSstLS0tLQorKDAgcm93cykKKworRFJPUCBUQUJMRSBqMSwgSjIsIEozOwpkaWZmIC0tZ2l0 IGEvc3JjL3Rlc3QvcmVncmVzcy9zcWwvam9pbi5zcWwgYi9zcmMvdGVzdC9yZWdyZXNzL3NxbC9q b2luLnNxbAppbmRleCBkODFmZjYzYmU1Li5mNDljNzMwZTU2IDEwMDY0NAotLS0gYS9zcmMvdGVz dC9yZWdyZXNzL3NxbC9qb2luLnNxbAorKysgYi9zcmMvdGVzdC9yZWdyZXNzL3NxbC9qb2luLnNx bApAQCAtMjk1MiwzICsyOTUyLDU3IEBAIFNFTEVDVCB0MS5hIEZST00gc2tpcF9mZXRjaCB0MSBM RUZUIEpPSU4gc2tpcF9mZXRjaCB0MiBPTiB0Mi5hID0gMSBXSEVSRSB0Mi5hIElTCiAKIFJFU0VU IGVuYWJsZV9pbmRleG9ubHlzY2FuOwogUkVTRVQgZW5hYmxlX3NlcXNjYW47CisKKy0tIFRlc3Qg VVNJTkcgd2l0aCBzeXN0ZW0gY29sdW1ucworCistLSBHZW5lcmF0ZSAyIHRhYmxlcyB3aXRoIDMg cm93cyBlYWNoLgorLS0geG1pbiAodHJhbnNhY3Rpb24gbnVtYmVyKSBpcyBhbiBpbnRlZ2VyIHZh bHVlIHJlcHJlc2VudGVkIGFzIFQxLCBUMiwgZXRjLgorLS0gVGhlIHhtaW4gdmFsdWVzIG9mIHRo ZSBmaXJzdCBhbmQgc2Vjb25kIHJvd3MgYXJlIGVxdWFsIHRvIGVhY2ggb3RoZXIsCistLSByZXNw ZWN0aXZlbHkuIFRoZSB2YWx1ZXMgVDMgYW5kIFQ0IG9mIHRoZSB0aGlyZCByb3dzIGFyZSBub3Qg ZXF1YWwuCistLSBKMTogaWQxIHwgeG1pbiAgICAgIEoyOiBpZDIgfCB4bWluCistLSAgICAgLS0t LSstLS0tLS0tICAgICAgIC0tLS0tKy0tLS0tLS0KKy0tICAgICAgIDEgfCBUMSAgICAgICAgICAg IDEwMSB8IFQxCistLSAgICAgICAyIHwgVDIgICAgICAgICAgICAxMDIgfCBUMgorLS0gICAgICAg MyB8IFQzICAgICAgICAgICAgMTAzIHwgVDQKK0NSRUFURSBUQUJMRSBqMSAoaWQxIElOVCk7CitD UkVBVEUgVEFCTEUgajIgKGlkMiBJTlQpOworQkVHSU47CisgIElOU0VSVCBJTlRPIGoxIChpZDEp IFZBTFVFUyAoMSk7CisgIElOU0VSVCBJTlRPIGoyIChpZDIpIFZBTFVFUyAoMTAxKTsKK0NPTU1J VDsKK0JFR0lOOworICBJTlNFUlQgSU5UTyBqMSAoaWQxKSBWQUxVRVMgKDIpOworICBJTlNFUlQg SU5UTyBqMiAoaWQyKSBWQUxVRVMgKDEwMik7CitDT01NSVQ7CitCRUdJTjsKKyAgSU5TRVJUIElO VE8gajEgKGlkMSkgVkFMVUVTICgzKTsKK0NPTU1JVDsKK0JFR0lOOworICBJTlNFUlQgSU5UTyBq MiAoaWQyKSBWQUxVRVMgKDEwMyk7CitDT01NSVQ7CisKKy0tIFRoZSB4bWluIHZhbHVlcyBhcmUg bm90IG91dHB1dCwgYXMgdGhleSB3aWxsIGRpZmZlciB3aXRoIGVhY2ggcnVuLgorLS0gU28sIG9u bHkgdGhlIGNvbHVtbiBoZWFkZXJzIGFuZCB0aGUgdmFsdWVzIG9mIGlkMSBhbmQgaWQyIGFyZSBk aXNwbGF5ZWQuCitTRUxFQ1QgKiBGUk9NIGoxIEpPSU4gajIgVVNJTkcgKHhtaW4pIExJTUlUIDA7 CitTRUxFQ1QgajEuaWQxLCBqMi5pZDIgRlJPTSBqMSBKT0lOIGoyIFVTSU5HICh4bWluKTsKK1NF TEVDVCAqIEZST00gajEgTEVGVCBKT0lOIGoyIFVTSU5HICh4bWluKSBMSU1JVCAwOworU0VMRUNU IGoxLmlkMSwgajIuaWQyIEZST00gajEgTEVGVCBKT0lOIGoyIFVTSU5HICh4bWluKTsKK1NFTEVD VCAqIEZST00gajEgUklHSFQgSk9JTiBqMiBVU0lORyAoeG1pbikgTElNSVQgMDsKK1NFTEVDVCBq MS5pZDEsIGoyLmlkMiBGUk9NIGoxIFJJR0hUIEpPSU4gajIgVVNJTkcgKHhtaW4pOworU0VMRUNU ICogRlJPTSBqMSBGVUxMIEpPSU4gajIgVVNJTkcgKHhtaW4pIExJTUlUIDA7CitTRUxFQ1QgajEu aWQxLCBqMi5pZDIgRlJPTSBqMSBGVUxMIEpPSU4gajIgVVNJTkcgKHhtaW4pOworLS0gVGVzdCBp ZiBVU0lORyBjYW4gYWRkIGFsbCBzeXN0ZW0gY29sdW1ucyBhdCBvbmNlCitTRUxFQ1QgKgorICBG Uk9NIGoxIEpPSU4gajIgVVNJTkcgKHRhYmxlb2lkLCB4bWluLCBjbWluLCB4bWF4LCBjbWF4LCBj dGlkKTsKKy0tIFRlc3QgVVNJTkcgZXhjZXB0aW9ucworU0VMRUNUICogRlJPTSBqMSBKT0lOIGoy IFVTSU5HIChpZDIpOworU0VMRUNUICogRlJPTSBqMSBKT0lOIGoyIFVTSU5HIChpZDEpOworV0lU SCBqMV9kdWJiZWQgQVMgKFNFTEVDVCBpZDEsIGlkMSBGUk9NIGoxKQorICBTRUxFQ1QgKiBGUk9N IGoxX2R1YmJlZCBKT0lOIGoxIFVTSU5HIChpZDEpOworV0lUSCBqMV9kdWJiZWQgQVMgKFNFTEVD VCBpZDEsIGlkMSBGUk9NIGoxKQorICBTRUxFQ1QgKiBGUk9NIGoxIEpPSU4gajFfZHViYmVkIFVT SU5HIChpZDEpOworLS0gVGVzdCBtdWx0aXBsZSBVU0lORyBqb2lucywgZWFjaCBiYXNlZCBvbiB0 aGUgcHJpb3IgcmVzdWx0LgorQ1JFQVRFIFRBQkxFIGozIChpZDMgSU5UKTsKK1NFTEVDVCAqIEZS T00gajEgSk9JTiBqMiBVU0lORyAoeG1pbikgSk9JTiBqMyBVU0lORyAoeG1pbikgTElNSVQgMDsK KworRFJPUCBUQUJMRSBqMSwgSjIsIEozOwotLSAKMi4zNC4xCgo= ------==--bound.60954.kxm5fi5dn62wfkjr.vla.yp-c.yandex.net-- Attachments: [text/x-diff] v1-0001-Add-system-column-support-to-the-USING-clause.patch (16.7K, ../../[email protected]/2-v1-0001-Add-system-column-support-to-the-USING-clause.patch) download | inline diff: From 295f9ec59b22cc3707fb8f168e0e3750e51724a0 Mon Sep 17 00:00:00 2001 From: Denis Garsh <[email protected]> Date: Wed, 7 Aug 2024 15:57:32 +0300 Subject: [PATCH] Add system column support to the USING clause Problem: The parser can't handle USING joins with system columns, such as xmin. Solution: 1. Use the scanNSItemForColumn() function instead of buildVarFromNSColumn() for constructing Var objects, as it correctly handles system columns. 2. Remove extra calls to markVarForSelectPriv(), since this function is already invoked in scanNSItemForColumn(). 3. For system columns, collect their negative attribute numbers along with user-defined column indices into l_colnos and r_colnos. 4. Create a buildVarFromSystemAttribute() function for rebuilding Var objects with system attributes, analogous to buildVarFromNSColumn(), since scanNSItemForColumn() is complex and has side effects. 5. Implement a fillNSColumnParametersFromVar() function for building NS columns with system attributes. 6. Add SystemAttributeTotalNumber() function to heap.c to ensure memory for res_nscolumns is allocated with system columns in mind. --- src/backend/catalog/heap.c | 8 ++ src/backend/parser/parse_clause.c | 178 ++++++++++++++++++++++++++--- src/include/catalog/heap.h | 2 + src/test/regress/expected/join.out | 106 +++++++++++++++++ src/test/regress/sql/join.sql | 54 +++++++++ 5 files changed, 330 insertions(+), 18 deletions(-) diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 01b43cc6a8..b44d0d0005 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -264,6 +264,14 @@ SystemAttributeByName(const char *attname) return NULL; } +/* + * Returns the total number of system attributes. + */ +size_t SystemAttributeTotalNumber(void) +{ + return lengthof(SysAtt); +} + /* ---------------------------------------------------------------- * XXX END OF UGLY HARD CODED BADNESS XXX diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index 8118036495..8e99d78272 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -24,6 +24,7 @@ #include "catalog/pg_amproc.h" #include "catalog/pg_constraint.h" #include "catalog/pg_type.h" +#include "catalog/heap.h" #include "commands/defrem.h" #include "miscadmin.h" #include "nodes/makefuncs.h" @@ -76,6 +77,10 @@ static Node *transformFromClauseItem(ParseState *pstate, Node *n, List **namespace); static Var *buildVarFromNSColumn(ParseState *pstate, ParseNamespaceColumn *nscol); +static Var *buildVarFromSystemAttribute(ParseState *pstate, + int rtindex, int attnum); +static void fillNSColumnParametersFromVar(ParseNamespaceColumn *rescolumn, + const Var *colvar); static Node *buildMergedJoinVar(ParseState *pstate, JoinType jointype, Var *l_colvar, Var *r_colvar); static void markRelsAsNulledBy(ParseState *pstate, Node *n, int jindex); @@ -273,7 +278,16 @@ extractRemainingColumns(ParseState *pstate, prevcols = NULL; foreach(lc, *src_colnos) { - prevcols = bms_add_member(prevcols, lfirst_int(lc)); + int col_index = lfirst_int(lc); + if (col_index < 0) + { + /* + * Attribute number of system column. Don't collect the number + * because the name of this column cannot be among src_colnames. + */ + continue; + } + prevcols = bms_add_member(prevcols, col_index); } attnum = 0; @@ -327,10 +341,6 @@ transformJoinUsingClause(ParseState *pstate, Var *rvar = (Var *) lfirst(rvars); A_Expr *e; - /* Require read access to the join variables */ - markVarForSelectPriv(pstate, lvar); - markVarForSelectPriv(pstate, rvar); - /* Now create the lvar = rvar join condition */ e = makeSimpleA_Expr(AEXPR_OP, "=", (Node *) copyObject(lvar), (Node *) copyObject(rvar), @@ -1291,7 +1301,9 @@ transformFromClauseItem(ParseState *pstate, Node *n, /* this may be larger than needed, but it's not worth being exact */ res_nscolumns = (ParseNamespaceColumn *) - palloc0((list_length(l_colnames) + list_length(r_colnames)) * + palloc0((list_length(l_colnames) + + list_length(r_colnames) + + SystemAttributeTotalNumber()) * sizeof(ParseNamespaceColumn)); res_colindex = 0; @@ -1332,6 +1344,14 @@ transformFromClauseItem(ParseState *pstate, Node *n, u_colname))); } + /* + * It is important to understand that among the l_colnames or + * r_colnames there may be a system column name. A system + * column name may appear if it was added in the previous step. + * This is because a query can contain multiple USING joins, + * where each join uses the result of the previous one. + */ + /* Find it in left input */ ndx = 0; foreach(col, l_colnames) @@ -1349,12 +1369,25 @@ transformFromClauseItem(ParseState *pstate, Node *n, } ndx++; } - if (l_index < 0) + + l_colvar = (Var *) scanNSItemForColumn(pstate, + l_nsitem, + 0, u_colname, -1); + if (l_colvar == NULL) ereport(ERROR, (errcode(ERRCODE_UNDEFINED_COLUMN), errmsg("column \"%s\" specified in USING clause does not exist in left table", u_colname))); - l_colnos = lappend_int(l_colnos, l_index + 1); + if (l_index >= 0) + { + /* User-defined column */ + l_colnos = lappend_int(l_colnos, l_index + 1); + } + else + { + /* System column */ + l_colnos = lappend_int(l_colnos, l_colvar->varattno); + } /* Find it in right input */ ndx = 0; @@ -1373,17 +1406,30 @@ transformFromClauseItem(ParseState *pstate, Node *n, } ndx++; } - if (r_index < 0) + + r_colvar = (Var *) scanNSItemForColumn(pstate, + r_nsitem, + 0, u_colname, -1); + + if (r_colvar == NULL) ereport(ERROR, (errcode(ERRCODE_UNDEFINED_COLUMN), errmsg("column \"%s\" specified in USING clause does not exist in right table", u_colname))); - r_colnos = lappend_int(r_colnos, r_index + 1); + + if (r_index >= 0) + { + /* User-defined column */ + r_colnos = lappend_int(r_colnos, r_index + 1); + } + else + { + /* System column */ + r_colnos = lappend_int(r_colnos, r_colvar->varattno); + } /* Build Vars to use in the generated JOIN ON clause */ - l_colvar = buildVarFromNSColumn(pstate, l_nscolumns + l_index); l_usingvars = lappend(l_usingvars, l_colvar); - r_colvar = buildVarFromNSColumn(pstate, r_nscolumns + r_index); r_usingvars = lappend(r_usingvars, r_colvar); /* @@ -1461,8 +1507,8 @@ transformFromClauseItem(ParseState *pstate, Node *n, /* Scan the colnos lists to recover info from the previous loop */ forboth(lc1, l_colnos, lc2, r_colnos) { - int l_index = lfirst_int(lc1) - 1; - int r_index = lfirst_int(lc2) - 1; + int l_index = lfirst_int(lc1); + int r_index = lfirst_int(lc2); Var *l_colvar, *r_colvar; Node *u_colvar; @@ -1472,8 +1518,33 @@ transformFromClauseItem(ParseState *pstate, Node *n, * Note we re-build these Vars: they might have different * varnullingrels than the ones made in the previous loop. */ - l_colvar = buildVarFromNSColumn(pstate, l_nscolumns + l_index); - r_colvar = buildVarFromNSColumn(pstate, r_nscolumns + r_index); + if (l_index > 0) + { + /* User-defined column */ + l_colvar = buildVarFromNSColumn(pstate, + l_nscolumns + l_index - 1); + } + else + { + /* System column, so l_index is attnum. */ + l_colvar = buildVarFromSystemAttribute(pstate, + l_nsitem->p_rtindex, + l_index); + } + + if (r_index > 0) + { + /* User-defined column */ + r_colvar = buildVarFromNSColumn(pstate, + r_nscolumns + r_index - 1); + } + else + { + /* System column, so r_index is attnum. */ + r_colvar = buildVarFromSystemAttribute(pstate, + r_nsitem->p_rtindex, + r_index); + } /* Construct the join alias Var for this column */ u_colvar = buildMergedJoinVar(pstate, @@ -1488,12 +1559,32 @@ transformFromClauseItem(ParseState *pstate, Node *n, if (u_colvar == (Node *) l_colvar) { /* Merged column is equivalent to left input */ - *res_nscolumn = l_nscolumns[l_index]; + if (l_index > 0) + { + /* User-defined column, so just take it. */ + *res_nscolumn = l_nscolumns[l_index - 1]; + } + else + { + /* System column, so construct new column. */ + fillNSColumnParametersFromVar(res_nscolumn, + l_colvar); + } } else if (u_colvar == (Node *) r_colvar) { /* Merged column is equivalent to right input */ - *res_nscolumn = r_nscolumns[r_index]; + if (r_index > 0) + { + /* User-defined column, so just take it. */ + *res_nscolumn = r_nscolumns[r_index - 1]; + } + else + { + /* System column, so construct new column. */ + fillNSColumnParametersFromVar(res_nscolumn, + r_colvar); + } } else { @@ -1658,6 +1749,57 @@ buildVarFromNSColumn(ParseState *pstate, ParseNamespaceColumn *nscol) return var; } +/* + * buildVarFromSystemAttribute - + * build a Var node using system attribute, e.g. "xmin". + * + * rtindex is the relation's index in the rangetable. + * attnum is the number of system attribute. + * Returns NULL if there is no such system attribute. + */ +static Var * +buildVarFromSystemAttribute(ParseState *pstate, int rtindex, int attnum) +{ + Var *var; + const FormData_pg_attribute *sysatt; + + sysatt = SystemAttributeDefinition(attnum); + + if (sysatt == NULL) + return NULL; + + var = makeVar(rtindex, + attnum, + sysatt->atttypid, + sysatt->atttypmod, + sysatt->attcollation, + 0); + + /* update varnullingrels */ + markNullableIfNeeded(pstate, var); + return var; +} + +/* + * fillNSColumnParametersFromVar - + * fill ParseNamespaceColumn data using Var data. + * + * rescolumn is pointer to the structure to be filled. + * colvar is the Var where to get data from. + */ +static void +fillNSColumnParametersFromVar(ParseNamespaceColumn *rescolumn, + const Var *colvar) +{ + rescolumn->p_varno = colvar->varno; + rescolumn->p_varattno = colvar->varattno; + rescolumn->p_vartype = colvar->vartype; + rescolumn->p_vartypmod = colvar->vartypmod; + rescolumn->p_varcollid = colvar->varcollid; + rescolumn->p_varnosyn = colvar->varnosyn; + rescolumn->p_varattnosyn = colvar->varattnosyn; +} + /* * buildMergedJoinVar - * generate a suitable replacement expression for a merged join column diff --git a/src/include/catalog/heap.h b/src/include/catalog/heap.h index c512824cd1..960db152b5 100644 --- a/src/include/catalog/heap.h +++ b/src/include/catalog/heap.h @@ -136,6 +136,8 @@ extern const FormData_pg_attribute *SystemAttributeDefinition(AttrNumber attno); extern const FormData_pg_attribute *SystemAttributeByName(const char *attname); +extern size_t SystemAttributeTotalNumber(void); + extern void CheckAttributeNamesTypes(TupleDesc tupdesc, char relkind, int flags); diff --git a/src/test/regress/expected/join.out b/src/test/regress/expected/join.out index 31fb7d142e..1ea9a1ab71 100644 --- a/src/test/regress/expected/join.out +++ b/src/test/regress/expected/join.out @@ -8017,3 +8017,109 @@ SELECT t1.a FROM skip_fetch t1 LEFT JOIN skip_fetch t2 ON t2.a = 1 WHERE t2.a IS RESET enable_indexonlyscan; RESET enable_seqscan; +-- Test USING with system columns +-- Generate 2 tables with 3 rows each. +-- xmin (transaction number) is an integer value represented as T1, T2, etc. +-- The xmin values of the first and second rows are equal to each other, +-- respectively. The values T3 and T4 of the third rows are not equal. +-- J1: id1 | xmin J2: id2 | xmin +-- ----+------- -----+------- +-- 1 | T1 101 | T1 +-- 2 | T2 102 | T2 +-- 3 | T3 103 | T4 +CREATE TABLE j1 (id1 INT); +CREATE TABLE j2 (id2 INT); +BEGIN; + INSERT INTO j1 (id1) VALUES (1); + INSERT INTO j2 (id2) VALUES (101); +COMMIT; +BEGIN; + INSERT INTO j1 (id1) VALUES (2); + INSERT INTO j2 (id2) VALUES (102); +COMMIT; +BEGIN; + INSERT INTO j1 (id1) VALUES (3); +COMMIT; +BEGIN; + INSERT INTO j2 (id2) VALUES (103); +COMMIT; +-- The xmin values are not output, as they will differ with each run. +-- So, only the column headers and the values of id1 and id2 are displayed. +SELECT * FROM j1 JOIN j2 USING (xmin) LIMIT 0; + xmin | id1 | id2 +------+-----+----- +(0 rows) + +SELECT j1.id1, j2.id2 FROM j1 JOIN j2 USING (xmin); + id1 | id2 +-----+----- + 1 | 101 + 2 | 102 +(2 rows) + +SELECT * FROM j1 LEFT JOIN j2 USING (xmin) LIMIT 0; + xmin | id1 | id2 +------+-----+----- +(0 rows) + +SELECT j1.id1, j2.id2 FROM j1 LEFT JOIN j2 USING (xmin); + id1 | id2 +-----+----- + 1 | 101 + 2 | 102 + 3 | +(3 rows) + +SELECT * FROM j1 RIGHT JOIN j2 USING (xmin) LIMIT 0; + xmin | id1 | id2 +------+-----+----- +(0 rows) + +SELECT j1.id1, j2.id2 FROM j1 RIGHT JOIN j2 USING (xmin); + id1 | id2 +-----+----- + 1 | 101 + 2 | 102 + | 103 +(3 rows) + +SELECT * FROM j1 FULL JOIN j2 USING (xmin) LIMIT 0; + xmin | id1 | id2 +------+-----+----- +(0 rows) + +SELECT j1.id1, j2.id2 FROM j1 FULL JOIN j2 USING (xmin); + id1 | id2 +-----+----- + 1 | 101 + 2 | 102 + 3 | + | 103 +(4 rows) + +-- Test if USING can add all system columns at once +SELECT * + FROM j1 JOIN j2 USING (tableoid, xmin, cmin, xmax, cmax, ctid); + tableoid | xmin | cmin | xmax | cmax | ctid | id1 | id2 +----------+------+------+------+------+------+-----+----- +(0 rows) + +-- Test USING exceptions +SELECT * FROM j1 JOIN j2 USING (id2); +ERROR: column "id2" specified in USING clause does not exist in left table +SELECT * FROM j1 JOIN j2 USING (id1); +ERROR: column "id1" specified in USING clause does not exist in right table +WITH j1_dubbed AS (SELECT id1, id1 FROM j1) + SELECT * FROM j1_dubbed JOIN j1 USING (id1); +ERROR: common column name "id1" appears more than once in left table +WITH j1_dubbed AS (SELECT id1, id1 FROM j1) + SELECT * FROM j1 JOIN j1_dubbed USING (id1); +ERROR: common column name "id1" appears more than once in right table +-- Test multiple USING joins, each based on the prior result. +CREATE TABLE j3 (id3 INT); +SELECT * FROM j1 JOIN j2 USING (xmin) JOIN j3 USING (xmin) LIMIT 0; + xmin | id1 | id2 | id3 +------+-----+-----+----- +(0 rows) + +DROP TABLE j1, J2, J3; diff --git a/src/test/regress/sql/join.sql b/src/test/regress/sql/join.sql index d81ff63be5..f49c730e56 100644 --- a/src/test/regress/sql/join.sql +++ b/src/test/regress/sql/join.sql @@ -2952,3 +2952,57 @@ SELECT t1.a FROM skip_fetch t1 LEFT JOIN skip_fetch t2 ON t2.a = 1 WHERE t2.a IS RESET enable_indexonlyscan; RESET enable_seqscan; + +-- Test USING with system columns + +-- Generate 2 tables with 3 rows each. +-- xmin (transaction number) is an integer value represented as T1, T2, etc. +-- The xmin values of the first and second rows are equal to each other, +-- respectively. The values T3 and T4 of the third rows are not equal. +-- J1: id1 | xmin J2: id2 | xmin +-- ----+------- -----+------- +-- 1 | T1 101 | T1 +-- 2 | T2 102 | T2 +-- 3 | T3 103 | T4 +CREATE TABLE j1 (id1 INT); +CREATE TABLE j2 (id2 INT); +BEGIN; + INSERT INTO j1 (id1) VALUES (1); + INSERT INTO j2 (id2) VALUES (101); +COMMIT; +BEGIN; + INSERT INTO j1 (id1) VALUES (2); + INSERT INTO j2 (id2) VALUES (102); +COMMIT; +BEGIN; + INSERT INTO j1 (id1) VALUES (3); +COMMIT; +BEGIN; + INSERT INTO j2 (id2) VALUES (103); +COMMIT; + +-- The xmin values are not output, as they will differ with each run. +-- So, only the column headers and the values of id1 and id2 are displayed. +SELECT * FROM j1 JOIN j2 USING (xmin) LIMIT 0; +SELECT j1.id1, j2.id2 FROM j1 JOIN j2 USING (xmin); +SELECT * FROM j1 LEFT JOIN j2 USING (xmin) LIMIT 0; +SELECT j1.id1, j2.id2 FROM j1 LEFT JOIN j2 USING (xmin); +SELECT * FROM j1 RIGHT JOIN j2 USING (xmin) LIMIT 0; +SELECT j1.id1, j2.id2 FROM j1 RIGHT JOIN j2 USING (xmin); +SELECT * FROM j1 FULL JOIN j2 USING (xmin) LIMIT 0; +SELECT j1.id1, j2.id2 FROM j1 FULL JOIN j2 USING (xmin); +-- Test if USING can add all system columns at once +SELECT * + FROM j1 JOIN j2 USING (tableoid, xmin, cmin, xmax, cmax, ctid); +-- Test USING exceptions +SELECT * FROM j1 JOIN j2 USING (id2); +SELECT * FROM j1 JOIN j2 USING (id1); +WITH j1_dubbed AS (SELECT id1, id1 FROM j1) + SELECT * FROM j1_dubbed JOIN j1 USING (id1); +WITH j1_dubbed AS (SELECT id1, id1 FROM j1) + SELECT * FROM j1 JOIN j1_dubbed USING (id1); +-- Test multiple USING joins, each based on the prior result. +CREATE TABLE j3 (id3 INT); +SELECT * FROM j1 JOIN j2 USING (xmin) JOIN j3 USING (xmin) LIMIT 0; + +DROP TABLE j1, J2, J3; -- 2.34.1 ^ permalink raw reply [nested|flat] 3+ messages in thread
* Re: Add system column support to the USING clause @ 2024-09-13 14:06 David G. Johnston <[email protected]> parent: Denis Garsh <[email protected]> 0 siblings, 0 replies; 3+ messages in thread From: David G. Johnston @ 2024-09-13 14:06 UTC (permalink / raw) To: Denis Garsh <[email protected]>; +Cc: [email protected] <[email protected]> On Friday, September 13, 2024, Denis Garsh <[email protected]> wrote: > > > The patch adds support for system columns in JOIN USING clause. > Definitely not high on my list of oversights to fix. Resorting to the ON clause for the rare query that would need to do such a thing isn’t that costly. But as the patch exists I’ll leave it to others to judge the cost of actually adding it, or worthwhile-mess of reviewing it. > Link to PR on GitHub: https://github.com/hilltracer/postgres/pull/3 > You apparently missed the note on GitHub that says we don’t work with pull requests. Patches are to be submitted directly to the mailing list. David J. ^ permalink raw reply [nested|flat] 3+ messages in thread
end of thread, other threads:[~2024-09-13 14:06 UTC | newest] Thread overview: 3+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2021-02-07 15:10 [PATCH v4 1/2] Add 'noError' argument to encoding conversion functions. Heikki Linnakangas <[email protected]> 2024-09-13 08:03 Add system column support to the USING clause Denis Garsh <[email protected]> 2024-09-13 14:06 ` Re: Add system column support to the USING clause David G. Johnston <[email protected]>
This inbox is served by agora; see mirroring instructions for how to clone and mirror all data and code used for this inbox