Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.98.2) (envelope-from ) id 1xOxTo-00000000Q8G-1E17 for pgsql-bugs@arkaria.postgresql.org; Thu, 05 Nov 2026 13:26:13 +0000 Received: from localhost ([127.0.0.1]:44152 helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.98.2) (envelope-from ) id 1xOxTn-0000000CYMA-43EN for pgsql-bugs@arkaria.postgresql.org; Thu, 05 Nov 2026 13:26:11 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]:49986) by malur.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.98.2) (envelope-from ) id 1xOtqG-0000000Bo9K-1eJU for pgsql-bugs@lists.postgresql.org; Thu, 05 Nov 2026 09:33:08 +0000 Received: from mahout.postgresql.org ([2001:4800:3e1:1::227]) by makus.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.98.2) (envelope-from ) id 1xOtqD-00000000L5M-2mAM for pgsql-bugs@lists.postgresql.org; Thu, 05 Nov 2026 09:33:07 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=postgresql.org; s=20171124; h=Message-ID:Date:Reply-To:Cc:From:To:Subject: Content-Transfer-Encoding:MIME-Version:Content-Type:Sender:Content-ID: Content-Description:In-Reply-To:References; bh=fUIRyFNlaekTvT6KkDAXLe4q0HJXpjYiuoz2vJHe40k=; b=J+Ly//cYQ3sQ/BHEaDUNqevSAY SZva7jbJwCDPcdH1JJHqOCKOCOSrgy+Tlzb8BxfwutDyhD/CH9PjKyxM0CexSu3pT/vYmmtdwRMRA SeRQJE/NgylRyXLQPSzx+GalBB9Iu58qK4FnxeE3/Asd4/CsEracd8HZkILuNaJHR1C5jcbBDx67r Rh/GYpQOpAd0Kk2D4ZDf1LF15JAjfPofK117GgEPclRacno7/ui1RbnrU4rlU0VbXD2yZevWXMx8j 2VOhWtZafBujIJK+IawKr5eWYzTAjhd6Rw7smUQ6ZBTmrDytehWqqB42XN+7veSHmUD9q3keMmE4+ C5A/nLgQ==; Received: from wrigleys.postgresql.org ([2a02:16a8:dc51::60]) by mahout.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.98.2) (envelope-from ) id 1xOtqC-00000000vrP-2wsu for pgsql-bugs@lists.postgresql.org; Thu, 05 Nov 2026 09:33:06 +0000 Received: from localhost ([127.0.0.1] helo=wrigleys.postgresql.org) by wrigleys.postgresql.org with esmtp (Exim 4.98.2) (envelope-from ) id 1xOtqB-00000002URl-2QsY for pgsql-bugs@lists.postgresql.org; Thu, 05 Nov 2026 09:33:04 +0000 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Subject: BUG #19832: convert() between BIG5 and EUC_TW returns different characters than PostgreSQL's own UTF-8 mappings To: pgsql-bugs@lists.postgresql.org From: PG Bug reporting form Cc: 303677365@qq.com Reply-To: 303677365@qq.com, pgsql-bugs@lists.postgresql.org Date: Thu, 05 Nov 2026 09:32:54 +0000 Message-ID: <19832-7f2119af7a736b33@postgresql.org> X-Auto-Response-Suppress: All Auto-Submitted: auto-generated List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk The following bug has been logged on the website: Bug reference: 19832 Logged by: chunling qin Email address: 303677365@qq.com PostgreSQL version: 18.6 Operating system: Linux x86_64 (official Docker image postgres:lates Description: =20 **Description:** This is the same family of problem as bug #19831 (single-byte Cyrillic conversions), but in the multi-byte BIG5 <-> EUC_TW pair: `convert()` takes a *direct* conversion path (the BIG5<->CNS 11643 tables registered in `pg_conversion`), while `convert_to()`/`convert_from()` go through UTF-8 using the Unicode-derived generated maps. For 22 byte sequences the two paths decode to *different characters*. One caveat up front: I am not aware of an independent third-party EUC_TW oracle to adjudicate which character is "correct" for a given BIG5 code. The evidence below is therefore framed as an internal inconsistency: two code paths inside PostgreSQL itself, both claiming to convert between the same two encodings, return different characters, and the swapped-pair structure (below) points to entries in the direct tables being shifted, rather than a legitimate mapping difference. **Reproduction (self-contained, works on the official image):** ```sh docker exec -i psql -U postgres -X -P pager=3Do= ff <<'SQL' -- BIG5 0xa1 0x55 is U+FF5C (FULLWIDTH VERTICAL LINE) per PG's own big5_to_utf8.map SELECT convert_from(decode('a155','hex'), 'BIG5') AS src_char, convert_from(convert(decode('a155','hex'),'BIG5','EUC_TW'),'EUC_TW') AS direct_char, convert_from(convert_to(convert_from(decode('a155','hex'),'BIG5'), 'EUC_TW'),'EUC_TW') AS twostep_char; -- direct_char =3D =EF=B8=B1 (U+FE31 PRESENTATION FORM FOR VERTICAL EM DASH) -- twostep_char =3D =EF=BD=9C (U+FF5C) =3D src_char SQL ``` **Forward direction (BIG5 -> EUC_TW), 12 cases; in every case `direct_char` is a different character from `src_char`, while `twostep_char` (PG's own UTF-8 maps) equals `src_char`:** | BIG5 | src_char | direct result | twostep result | |---|---|---|---| | a155 | =EF=BD=9C U+FF5C | a1b6 =3D =EF=B8=B1 U+FE31 | a2de =3D =EF=BD=9C | | a156 | =E2=80=93 U+2013 | a1b7 =3D =E2=80=94 U+2014 | a1b9 =3D =E2=80=93 | | a157 | =EF=B8=B1 U+FE31 | a1b8 =3D =EF=B8=B2 | a1b6 =3D =EF=B8=B1 | | a158 | =E2=80=94 U+2014 | a1b9 =3D =E2=80=93 U+2013 | a1a7 =3D =E2=80=94 | | a1ab | =E2=80=B5 U+2035 | a1ea =3D =E2=80=B2 U+2032 | a1eb =3D =E2=80=B5 | | a1ac | =E2=80=B2 U+2032 | a1eb =3D =E2=80=B5 U+2035 | a1ea =3D =E2=80=B2 | | a1e1 | =EF=B9=A5 | a2c2 =3D =EF=B9=A6 | a2c3 =3D =EF=B9=A5 | | a1e2 | =EF=B9=A6 | a2c3 =3D =EF=B9=A5 | a2c2 =3D =EF=B9=A6 | | a241 | =EF=BC=8F U+FF0F | a2e1 =3D =E2=88=95 U+2215 | a2df =3D =EF=BC=8F | | a242 | =EF=BC=BC U+FF3C | a2e2 =3D =EF=B9=A8 | a2e0 =3D =EF=BC=BC | | d6cc | =E7=AD=87 | 8ea2b7ef =3D =E7=AC=BB | 8ea2b7ef... =3D =E7=AD=87 | | dadf | =E7=AC=BB | 8ea2bee3 =3D =E7=AD=87 | 8ea2bee3... =3D =E7=AC=BB | (For the last two, the CNS-plane encodings are shown as EUC_TW byte sequences; direct and two-step swap the two CJK characters.) The structure is striking: the direct result of input X is the character that PG's own maps assign to input Y, and vice versa =E2=80=94 a155<->a157, a156<->a158, a1ab<->a1ac, a1e1<->a1e2, d6cc<->dadf. The direct BIG5->CNS entries look **shifted/swapped by one position** relative to the Unicode-derived maps, which is a typical transcription bug pattern rather than a mapping-convention difference. **Reverse direction (EUC_TW -> BIG5):** 8 further cases show the same disagreement and the same swapped-pair structure (a1b6/a1b7, a1ea/a1eb, a2c2/a2c3, a2de/a2df ...). Two additional reverse cases (a2df/a2e0) involve U+FFFD replacement-character entries in `big5_to_utf8.map` (a known separate mapping difference) and are excluded from this report. **Root cause:** `src/backend/utils/mb/conversion_procs/euc_tw_and_big5/big5.c` =E2=80=94 `BIG5toCNS()` / `CNStoBIG5()` use legacy hand-maintained BIG5<->CNS 11643 tables, whose entries disagree for the byte pairs above with the Unicode-derived generated maps `src/backend/utils/mb/Unicode/big5_to_utf8.map` (from `UCS_to_BIG5.pl`) and `euc_tw_to_utf8.map` / `utf8_to_euc_tw.map` (from `UCS_to_EUC_TW.pl`). **Expected behavior:** as with #19831, the direct conversion and the composition of PG's own Unicode-derived maps should agree (or the direct tables should be dropped in favor of routing through UTF-8). Given the swapped-pair structure, the most likely fix is correcting the shifted entries in the BIG5<->CNS tables; if instead the legacy tables are considered authoritative for these pairs, then PG's two code paths will keep giving different answers for the same conversion, which is at minimum a consistency defect.