Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1n54v9-00076s-5H for pgsql-hackers@arkaria.postgresql.org; Wed, 05 Jan 2022 11:57:35 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1n54v5-0006Lg-W6 for pgsql-hackers@arkaria.postgresql.org; Wed, 05 Jan 2022 11:57:31 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1n54v5-0006LX-EQ for pgsql-hackers@lists.postgresql.org; Wed, 05 Jan 2022 11:57:31 +0000 Received: from sid.nimrod.no ([2001:8c0:9340:1::2]) by makus.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1n54uy-0008Ms-4g for pgsql-hackers@lists.postgresql.org; Wed, 05 Jan 2022 11:57:30 +0000 Received: from sid.nimrod.no (sid.nimrod.no [195.139.160.2]) by sid.nimrod.no (8.14.7/8.14.7) with ESMTP id 205BvLrv031638; Wed, 5 Jan 2022 12:57:21 +0100 DKIM-Filter: OpenDKIM Filter v2.11.0 sid.nimrod.no 205BvLrv031638 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nimrod.no; s=n20181209; t=1641383842; bh=R9Qks7p1/XCQ29gqEO1XU9VkcmPG+YYQK8Ix7dkFfR0=; h=From:To:Cc:Subject:References:Date:In-Reply-To:From; b=x+kCu+ZxkiL4Zfd/DcfTNXelYaNDOBvv/u4ArHjgXiMbq1Gq/9KxlRfMYZ1t/y2+G NnRcmUY6mKcc/Rk+KaddTFugDAGsVM+RjQQrm/UlZEzoof8MgDonPHa8rdNPhZbs+L F8351tco7RmRrwLVwEXiFwyHc9RP8HOc31SXVjs5D4/r9b+zxa6EjXwGUrSWYqoWXO 1kngqIo4Mqh6tSKAiuDFwUMk6Y072sF5ZhX2Zq1I+fh7GjYXi0ch6b94Kkpf2o5/Dz Is1mx3jq6QofFXYb61WtqzVcFMKlxQbr7RQ9IK0oNv0nRvovcF0CLBZ3ZHTnR9JP2Q 2jpZ1Bb0ieJuw== From: Dag Lem To: Tom Lane Cc: PostgreSQL Hackers Subject: [PATCH] Run UTF8-dependent tests for citext [Re: daitch_mokotoff module] Organization: Nimrod References: <0cac2d60-5b85-59c6-1ac1-77092b8688b7@enterprisedb.com> <20220102213150.y2tpuwqhruu5t2y5@alap3.anarazel.de> <3461033.1641177713@sss.pgh.pa.us> <3563190.1641227676@sss.pgh.pa.us> Date: Wed, 05 Jan 2022 12:57:21 +0100 In-Reply-To: <3563190.1641227676@sss.pgh.pa.us> (Tom Lane's message of "Mon, 03 Jan 2022 11:34:36 -0500") Message-ID: User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.3 (gnu/linux) MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk --=-=-= Content-Type: text/plain Tom Lane writes: > Dag Lem writes: > >> Running "ack -l '[\x80-\xff]'" in the contrib/ directory reveals that >> two other modules are using UTF8 characters in tests - citext and >> unaccent. > > Yeah, neither of those have been upgraded to said best practice. > (If you feel like doing the legwork to improve that situation, > that'd be great.) > Please find attached a patch to run the previously commented-out UTF8-dependent tests for citext, according to best practice. For now I don't dare to touch the unaccent module, which seems to be UTF8-only anyway. Best regards Dag Lem --=-=-= Content-Type: text/x-patch; charset=utf-8 Content-Disposition: inline; filename=citext-utf8-tests.patch Content-Transfer-Encoding: quoted-printable diff --git a/contrib/citext/Makefile b/contrib/citext/Makefile index a7de52928d..789932fe36 100644 --- a/contrib/citext/Makefile +++ b/contrib/citext/Makefile @@ -11,7 +11,7 @@ DATA =3D citext--1.4.sql \ citext--1.0--1.1.sql PGFILEDESC =3D "citext - case-insensitive character string data type" =20 -REGRESS =3D citext +REGRESS =3D citext citext_utf8 =20 ifdef USE_PGXS PG_CONFIG =3D pg_config diff --git a/contrib/citext/expected/citext.out b/contrib/citext/expected/c= itext.out index 3bac0534fb..48b4de8993 100644 --- a/contrib/citext/expected/citext.out +++ b/contrib/citext/expected/citext.out @@ -48,29 +48,6 @@ SELECT 'a'::citext <> 'ab'::citext AS t; t (1 row) =20 --- Multibyte sanity tests. Uncomment to run. --- SELECT '=C3=80'::citext =3D '=C3=80'::citext AS t; --- SELECT '=C3=80'::citext =3D '=C3=A0'::citext AS t; --- SELECT '=C3=80'::text =3D '=C3=A0'::text AS f; -- text wins. --- SELECT '=C3=80'::citext <> 'B'::citext AS t; --- Test combining characters making up canonically equivalent strings. --- SELECT '=C3=84'::text <> 'A=CC=88'::text AS t; --- SELECT '=C3=84'::citext <> 'A=CC=88'::citext AS t; --- Test the Turkish dotted I. The lowercase is a single byte while the --- uppercase is multibyte. This is why the comparison code can't be optimi= zed --- to compare string lengths. --- SELECT 'i'::citext =3D '=C4=B0'::citext AS t; --- Regression. --- SELECT 'l=C3=A1ska'::citext <> 'lask=C3=A1'::citext AS t; --- SELECT 'Ask Bj=C3=B8rn Hansen'::citext =3D 'Ask Bj=C3=B8rn Hansen'::cit= ext AS t; --- SELECT 'Ask Bj=C3=B8rn Hansen'::citext =3D 'ASK BJ=C3=98RN HANSEN'::cit= ext AS t; --- SELECT 'Ask Bj=C3=B8rn Hansen'::citext <> 'Ask Bjorn Hansen'::citext AS= t; --- SELECT 'Ask Bj=C3=B8rn Hansen'::citext <> 'ASK BJORN HANSEN'::citext AS= t; --- SELECT citext_cmp('Ask Bj=C3=B8rn Hansen'::citext, 'Ask Bj=C3=B8rn Hans= en'::citext) AS zero; --- SELECT citext_cmp('Ask Bj=C3=B8rn Hansen'::citext, 'ask bj=C3=B8rn hans= en'::citext) AS zero; --- SELECT citext_cmp('Ask Bj=C3=B8rn Hansen'::citext, 'ASK BJ=C3=98RN HANS= EN'::citext) AS zero; --- SELECT citext_cmp('Ask Bj=C3=B8rn Hansen'::citext, 'Ask Bjorn Hansen'::= citext) AS positive; --- SELECT citext_cmp('Ask Bjorn Hansen'::citext, 'Ask Bj=C3=B8rn Hansen'::= citext) AS negative; -- Test > and >=3D SELECT 'B'::citext > 'a'::citext AS t; t=20 diff --git a/contrib/citext/expected/citext_1.out b/contrib/citext/expected= /citext_1.out index 57fc863f7a..8ab4d4224e 100644 --- a/contrib/citext/expected/citext_1.out +++ b/contrib/citext/expected/citext_1.out @@ -48,29 +48,6 @@ SELECT 'a'::citext <> 'ab'::citext AS t; t (1 row) =20 --- Multibyte sanity tests. Uncomment to run. --- SELECT '=C3=80'::citext =3D '=C3=80'::citext AS t; --- SELECT '=C3=80'::citext =3D '=C3=A0'::citext AS t; --- SELECT '=C3=80'::text =3D '=C3=A0'::text AS f; -- text wins. --- SELECT '=C3=80'::citext <> 'B'::citext AS t; --- Test combining characters making up canonically equivalent strings. --- SELECT '=C3=84'::text <> 'A=CC=88'::text AS t; --- SELECT '=C3=84'::citext <> 'A=CC=88'::citext AS t; --- Test the Turkish dotted I. The lowercase is a single byte while the --- uppercase is multibyte. This is why the comparison code can't be optimi= zed --- to compare string lengths. --- SELECT 'i'::citext =3D '=C4=B0'::citext AS t; --- Regression. --- SELECT 'l=C3=A1ska'::citext <> 'lask=C3=A1'::citext AS t; --- SELECT 'Ask Bj=C3=B8rn Hansen'::citext =3D 'Ask Bj=C3=B8rn Hansen'::cit= ext AS t; --- SELECT 'Ask Bj=C3=B8rn Hansen'::citext =3D 'ASK BJ=C3=98RN HANSEN'::cit= ext AS t; --- SELECT 'Ask Bj=C3=B8rn Hansen'::citext <> 'Ask Bjorn Hansen'::citext AS= t; --- SELECT 'Ask Bj=C3=B8rn Hansen'::citext <> 'ASK BJORN HANSEN'::citext AS= t; --- SELECT citext_cmp('Ask Bj=C3=B8rn Hansen'::citext, 'Ask Bj=C3=B8rn Hans= en'::citext) AS zero; --- SELECT citext_cmp('Ask Bj=C3=B8rn Hansen'::citext, 'ask bj=C3=B8rn hans= en'::citext) AS zero; --- SELECT citext_cmp('Ask Bj=C3=B8rn Hansen'::citext, 'ASK BJ=C3=98RN HANS= EN'::citext) AS zero; --- SELECT citext_cmp('Ask Bj=C3=B8rn Hansen'::citext, 'Ask Bjorn Hansen'::= citext) AS positive; --- SELECT citext_cmp('Ask Bjorn Hansen'::citext, 'Ask Bj=C3=B8rn Hansen'::= citext) AS negative; -- Test > and >=3D SELECT 'B'::citext > 'a'::citext AS t; t=20 diff --git a/contrib/citext/expected/citext_utf8.out b/contrib/citext/expec= ted/citext_utf8.out new file mode 100644 index 0000000000..1f4fa79aff --- /dev/null +++ b/contrib/citext/expected/citext_utf8.out @@ -0,0 +1,119 @@ +/* + * This test must be run in a database with UTF-8 encoding, + * because other encodings don't support all the characters used. + */ +SELECT getdatabaseencoding() <> 'UTF8' + AS skip_test \gset +\if :skip_test +\quit +\endif +set client_encoding =3D utf8; +-- CREATE EXTENSION IF NOT EXISTS citext; +-- Multibyte sanity tests. +SELECT '=C3=80'::citext =3D '=C3=80'::citext AS t; + t=20 +--- + t +(1 row) + +SELECT '=C3=80'::citext =3D '=C3=A0'::citext AS t; + t=20 +--- + t +(1 row) + +SELECT '=C3=80'::text =3D '=C3=A0'::text AS f; -- text wins. + f=20 +--- + f +(1 row) + +SELECT '=C3=80'::citext <> 'B'::citext AS t; + t=20 +--- + t +(1 row) + +-- Test combining characters making up canonically equivalent strings. +SELECT '=C3=84'::text <> 'A=CC=88'::text AS t; + t=20 +--- + t +(1 row) + +SELECT '=C3=84'::citext <> 'A=CC=88'::citext AS t; + t=20 +--- + t +(1 row) + +-- Test the Turkish dotted I. The lowercase is a single byte while the +-- uppercase is multibyte. This is why the comparison code can't be optimi= zed +-- to compare string lengths. +SELECT 'i'::citext =3D '=C4=B0'::citext AS t; + t=20 +--- + t +(1 row) + +-- Regression. +SELECT 'l=C3=A1ska'::citext <> 'lask=C3=A1'::citext AS t; + t=20 +--- + t +(1 row) + +SELECT 'Ask Bj=C3=B8rn Hansen'::citext =3D 'Ask Bj=C3=B8rn Hansen'::citext= AS t; + t=20 +--- + t +(1 row) + +SELECT 'Ask Bj=C3=B8rn Hansen'::citext =3D 'ASK BJ=C3=98RN HANSEN'::citext= AS t; + t=20 +--- + t +(1 row) + +SELECT 'Ask Bj=C3=B8rn Hansen'::citext <> 'Ask Bjorn Hansen'::citext AS t; + t=20 +--- + t +(1 row) + +SELECT 'Ask Bj=C3=B8rn Hansen'::citext <> 'ASK BJORN HANSEN'::citext AS t; + t=20 +--- + t +(1 row) + +SELECT citext_cmp('Ask Bj=C3=B8rn Hansen'::citext, 'Ask Bj=C3=B8rn Hansen'= ::citext) AS zero; + zero=20 +------ + 0 +(1 row) + +SELECT citext_cmp('Ask Bj=C3=B8rn Hansen'::citext, 'ask bj=C3=B8rn hansen'= ::citext) AS zero; + zero=20 +------ + 0 +(1 row) + +SELECT citext_cmp('Ask Bj=C3=B8rn Hansen'::citext, 'ASK BJ=C3=98RN HANSEN'= ::citext) AS zero; + zero=20 +------ + 0 +(1 row) + +SELECT citext_cmp('Ask Bj=C3=B8rn Hansen'::citext, 'Ask Bjorn Hansen'::cit= ext) AS positive; + positive=20 +---------- + 15 +(1 row) + +SELECT citext_cmp('Ask Bjorn Hansen'::citext, 'Ask Bj=C3=B8rn Hansen'::cit= ext) AS negative; + negative=20 +---------- + -15 +(1 row) + diff --git a/contrib/citext/expected/citext_utf8_1.out b/contrib/citext/exp= ected/citext_utf8_1.out new file mode 100644 index 0000000000..37aead89c0 --- /dev/null +++ b/contrib/citext/expected/citext_utf8_1.out @@ -0,0 +1,8 @@ +/* + * This test must be run in a database with UTF-8 encoding, + * because other encodings don't support all the characters used. + */ +SELECT getdatabaseencoding() <> 'UTF8' + AS skip_test \gset +\if :skip_test +\quit diff --git a/contrib/citext/sql/citext.sql b/contrib/citext/sql/citext.sql index 55fb1d11a6..bd62ab8047 100644 --- a/contrib/citext/sql/citext.sql +++ b/contrib/citext/sql/citext.sql @@ -19,34 +19,6 @@ SELECT 'a'::citext =3D 'b'::citext AS f; SELECT 'a'::citext =3D 'ab'::citext AS f; SELECT 'a'::citext <> 'ab'::citext AS t; =20 --- Multibyte sanity tests. Uncomment to run. --- SELECT '=C3=80'::citext =3D '=C3=80'::citext AS t; --- SELECT '=C3=80'::citext =3D '=C3=A0'::citext AS t; --- SELECT '=C3=80'::text =3D '=C3=A0'::text AS f; -- text wins. --- SELECT '=C3=80'::citext <> 'B'::citext AS t; - --- Test combining characters making up canonically equivalent strings. --- SELECT '=C3=84'::text <> 'A=CC=88'::text AS t; --- SELECT '=C3=84'::citext <> 'A=CC=88'::citext AS t; - --- Test the Turkish dotted I. The lowercase is a single byte while the --- uppercase is multibyte. This is why the comparison code can't be optimi= zed --- to compare string lengths. --- SELECT 'i'::citext =3D '=C4=B0'::citext AS t; - --- Regression. --- SELECT 'l=C3=A1ska'::citext <> 'lask=C3=A1'::citext AS t; - --- SELECT 'Ask Bj=C3=B8rn Hansen'::citext =3D 'Ask Bj=C3=B8rn Hansen'::cit= ext AS t; --- SELECT 'Ask Bj=C3=B8rn Hansen'::citext =3D 'ASK BJ=C3=98RN HANSEN'::cit= ext AS t; --- SELECT 'Ask Bj=C3=B8rn Hansen'::citext <> 'Ask Bjorn Hansen'::citext AS= t; --- SELECT 'Ask Bj=C3=B8rn Hansen'::citext <> 'ASK BJORN HANSEN'::citext AS= t; --- SELECT citext_cmp('Ask Bj=C3=B8rn Hansen'::citext, 'Ask Bj=C3=B8rn Hans= en'::citext) AS zero; --- SELECT citext_cmp('Ask Bj=C3=B8rn Hansen'::citext, 'ask bj=C3=B8rn hans= en'::citext) AS zero; --- SELECT citext_cmp('Ask Bj=C3=B8rn Hansen'::citext, 'ASK BJ=C3=98RN HANS= EN'::citext) AS zero; --- SELECT citext_cmp('Ask Bj=C3=B8rn Hansen'::citext, 'Ask Bjorn Hansen'::= citext) AS positive; --- SELECT citext_cmp('Ask Bjorn Hansen'::citext, 'Ask Bj=C3=B8rn Hansen'::= citext) AS negative; - -- Test > and >=3D SELECT 'B'::citext > 'a'::citext AS t; SELECT 'b'::citext > 'A'::citext AS t; diff --git a/contrib/citext/sql/citext_utf8.sql b/contrib/citext/sql/citext= _utf8.sql new file mode 100644 index 0000000000..91822b85c2 --- /dev/null +++ b/contrib/citext/sql/citext_utf8.sql @@ -0,0 +1,42 @@ +/* + * This test must be run in a database with UTF-8 encoding, + * because other encodings don't support all the characters used. + */ + +SELECT getdatabaseencoding() <> 'UTF8' + AS skip_test \gset +\if :skip_test +\quit +\endif + +set client_encoding =3D utf8; + +-- CREATE EXTENSION IF NOT EXISTS citext; + +-- Multibyte sanity tests. +SELECT '=C3=80'::citext =3D '=C3=80'::citext AS t; +SELECT '=C3=80'::citext =3D '=C3=A0'::citext AS t; +SELECT '=C3=80'::text =3D '=C3=A0'::text AS f; -- text wins. +SELECT '=C3=80'::citext <> 'B'::citext AS t; + +-- Test combining characters making up canonically equivalent strings. +SELECT '=C3=84'::text <> 'A=CC=88'::text AS t; +SELECT '=C3=84'::citext <> 'A=CC=88'::citext AS t; + +-- Test the Turkish dotted I. The lowercase is a single byte while the +-- uppercase is multibyte. This is why the comparison code can't be optimi= zed +-- to compare string lengths. +SELECT 'i'::citext =3D '=C4=B0'::citext AS t; + +-- Regression. +SELECT 'l=C3=A1ska'::citext <> 'lask=C3=A1'::citext AS t; + +SELECT 'Ask Bj=C3=B8rn Hansen'::citext =3D 'Ask Bj=C3=B8rn Hansen'::citext= AS t; +SELECT 'Ask Bj=C3=B8rn Hansen'::citext =3D 'ASK BJ=C3=98RN HANSEN'::citext= AS t; +SELECT 'Ask Bj=C3=B8rn Hansen'::citext <> 'Ask Bjorn Hansen'::citext AS t; +SELECT 'Ask Bj=C3=B8rn Hansen'::citext <> 'ASK BJORN HANSEN'::citext AS t; +SELECT citext_cmp('Ask Bj=C3=B8rn Hansen'::citext, 'Ask Bj=C3=B8rn Hansen'= ::citext) AS zero; +SELECT citext_cmp('Ask Bj=C3=B8rn Hansen'::citext, 'ask bj=C3=B8rn hansen'= ::citext) AS zero; +SELECT citext_cmp('Ask Bj=C3=B8rn Hansen'::citext, 'ASK BJ=C3=98RN HANSEN'= ::citext) AS zero; +SELECT citext_cmp('Ask Bj=C3=B8rn Hansen'::citext, 'Ask Bjorn Hansen'::cit= ext) AS positive; +SELECT citext_cmp('Ask Bjorn Hansen'::citext, 'Ask Bj=C3=B8rn Hansen'::cit= ext) AS negative; --=-=-=--