Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.98.2) (envelope-from ) id 1xOxTo-00000000Q8F-1DUM for pgsql-bugs@arkaria.postgresql.org; Thu, 05 Nov 2026 13:26:13 +0000 Received: from localhost ([127.0.0.1]:44152 helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.98.2) (envelope-from ) id 1xOxTn-0000000CYMA-40PG for pgsql-bugs@arkaria.postgresql.org; Thu, 05 Nov 2026 13:26:11 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]:41110) by malur.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.98.2) (envelope-from ) id 1xOtLL-0000000BYSN-26Sb for pgsql-bugs@lists.postgresql.org; Thu, 05 Nov 2026 09:01:11 +0000 Received: from mahout.postgresql.org ([2001:4800:3e1:1::227]) by magus.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.98.2) (envelope-from ) id 1xOtLH-00000000NGo-2BO1 for pgsql-bugs@lists.postgresql.org; Thu, 05 Nov 2026 09:01:11 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=postgresql.org; s=20171124; h=Message-ID:Date:Reply-To:Cc:From:To:Subject: Content-Transfer-Encoding:MIME-Version:Content-Type:Sender:Content-ID: Content-Description:In-Reply-To:References; bh=Y8GQAvoVnz/SFlKdcO9Ez6pyjY25VdyDT+b3Usaqg/0=; b=aFQZcZP2N001nGnpsBd7tYwXd0 oD9SBs94BsfD/9ZTflEKxAabSE7K/dNy+iEEmnHtonArl7XQ0L11K/NCGIsAP0W9t0UQJQtaUgAZ9 Quj9QcE45AJXHkVN94edtOPYOkJlXwcR8JrhII0lsWRFPW0cS1qTEyTxgAJMrKE+7JdurZfIqObnB 38/zP2weEwP41TFWmeFI1CPTIS9YTs3ivbcw3DgwSHbHdh1gntZ3xMjmeo1amj6FUm/pSvJ6wDUGB r9aEfoiOaYEfps98u+VdvVqX2PdWko0lTjyD3wKujDucxDkfnU0aF9Ccf8IRM/IZRMJcaM67HXkmg h4SOEwAQ==; Received: from wrigleys.postgresql.org ([2a02:16a8:dc51::60]) by mahout.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.98.2) (envelope-from ) id 1xOtLE-00000000v9l-1wX6 for pgsql-bugs@lists.postgresql.org; Thu, 05 Nov 2026 09:01:06 +0000 Received: from localhost ([127.0.0.1] helo=wrigleys.postgresql.org) by wrigleys.postgresql.org with esmtp (Exim 4.98.2) (envelope-from ) id 1xOtLD-00000002SZs-1Q5V for pgsql-bugs@lists.postgresql.org; Thu, 05 Nov 2026 09:01:03 +0000 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Subject: BUG #19831: convert() between single-byte Cyrillic encodings returns wrong characters (KOI8R<->WIN866, WIN866->W To: pgsql-bugs@lists.postgresql.org From: PG Bug reporting form Cc: 303677365@qq.com Reply-To: 303677365@qq.com, pgsql-bugs@lists.postgresql.org Date: Thu, 05 Nov 2026 09:00:42 +0000 Message-ID: <19831-7ebbea4e9e6d9c34@postgresql.org> X-Auto-Response-Suppress: All Auto-Submitted: auto-generated List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk The following bug has been logged on the website: Bug reference: 19831 Logged by: chunling qin Email address: 303677365@qq.com PostgreSQL version: 18.6 Operating system: Linux x86_64 (official Docker image postgres:lates Description: =20 **Description:** `convert()` between two single-byte Cyrillic encodings takes a *direct* conversion path (a hand-written per-byte table registered in `pg_conversion`), while `convert_to()`/`convert_from()` always go through UTF-8 using the Unicode-derived generated maps. For several encoding pairs the two paths disagree: `convert()` silently returns a *different character*, returns a byte for a character that PostgreSQL itself says has no representation in the target encoding, or errors out for a character that is representable. This is not limited to the `convert()` function: the same direct conversions are used for automatic client/server encoding conversion and for `COPY ... ENCODING`. On a database created with `ENCODING 'KOI8R'`, I verified that `SET client_encoding=3D'WIN866'` returns corrupted results and that `COPY F= ROM ... ENCODING 'WIN866'` writes corrupted bytes to disk (details available on request). **Reproduction (self-contained, works on the official image):** ```sh docker exec -i psql -U postgres -X -P pager=3Do= ff <<'SQL' -- KOI8-R byte 0xa4 is U+2553 (BOX DRAWINGS DOUBLE VERTICAL AND HORIZONTAL) SELECT convert_from(decode('a4','hex'), 'KOI8R') AS src_char; SELECT encode(convert(decode('a4','hex'),'KOI8R','WIN866'), 'hex') AS direct; SELECT encode(convert_to(convert_from(decode('a4','hex'),'KOI8R'), 'WIN866'), 'hex') AS two_step_via_utf8; -- a longer string, showing silent character substitution: SELECT convert_from(decode('a441a6','hex'), 'KOI8R') AS original, convert_from(convert(decode('a441a6','hex'),'KOI8R','WIN866'), 'WIN866') AS converted; SQL ``` **Actual output:** ``` src_char | direct | two_step_via_utf8 ----------+--------+-------------------- =E2=95=93 | f3 | d6 original | converted ----------+----------- =E2=95=93A=E2=95=95 | =D1=94A=E2=88=99 ``` `f3` in WIN866 is U+0454 (CYRILLIC SMALL LETTER UKRAINIAN IE) =E2=80=94 a d= ifferent character. The two-step path through PostgreSQL's own UTF-8 maps yields `d6`, which is the correct byte for U+2553 in WIN866. **Full matrix (all verified on the official 18.6 image):** | Conversion | Input | convert() returns | Correct (per PG's own UTF-8 maps) | |---|---|---|---| | KOI8R -> WIN866 | a4 (U+2553) | f3 =3D =D1=94 U+0454 | d6 | | KOI8R -> WIN866 | a6 (U+2555) | f9 =3D =E2=88=99 U+2219 | b8 | | KOI8R -> WIN866 | a7 (U+2556) | f5 =3D =D1=97 U+0457 | b7 | | KOI8R -> WIN866 | ad (U+255C) | ad =3D =D0=BD U+043D | bd | | KOI8R -> WIN866 | b4/b6/b7/bd | f2/f8/f4/bd (wrong chars) | b6/d1/d2/d7 | | WIN866 -> KOI8R | bd/f8/f9 | bd/b6/a6 (wrong chars) | ad/9c/95 | | WIN866 -> WIN1251 | f8 (=C2=B0 U+00B0) | b2 =3D =D0=86 U+0406 | b0 | | KOI8R -> WIN1251 | a4 (U+2553) | **ba (no error!)** | should raise 22P05 (U+2553 has no WIN1251 representation; PG's own convert_to() does error) | | ISO_8859_5 -> WIN1251 | a0 (U+00A0 NBSP) | **ERROR 22P05** | a0 (WIN1251 does have NBSP at 0xa0; PG's own convert_to() succeeds) | | LATIN2 -> WIN1250 | 80 (U+0080) | **80 passed through =3D =E2=82=AC U+20A= C** | should raise 22P05 (U+0080 has no WIN1250 representation; PG's own convert_to() does error) | So the same bug family produces three distinct symptoms: 1. **Silent character substitution** (wrong result / data corruption), 2. **Missing error** where PostgreSQL's own conversion errors, 3. **False error** where PostgreSQL's own conversion succeeds. **Root cause:** `src/backend/utils/mb/conversion_procs/cyrillic/cyrillic.c` contains hand-written 128-entry tables (`koi2win866`, `win8662koi`, `koi2win1251`, `win12512koi`, `iso2win1251`, ...) consumed by `local2local()` (`src/backend/utils/mb/conv.c:33`), which indexes them with the low 7 bits of each byte. At least some entries disagree with the Unicode-derived generated maps (`src/backend/utils/mb/Unicode/*`). For example, in `koi2win866[]` the entry for index 0x24 (i.e. input byte 0xa4) is `0xF3`, while composing PG's own `koi8r_to_utf8` + `utf8_to_win866` maps requires `0xD6`. The tables appear to be historical hand-written data that was never audited against the Unicode-derived maps. The regression test `src/test/regress/sql/conversion.sql` covers only `iso8859-5 -> koi8r` with a few inputs (ASCII, Cyrillic letters, embedded NUL) and does not exercise the pseudographics byte range 0x80-0xBF where the disagreements live, which is presumably why this has gone unnoticed. **Expected behavior:** a direct conversion between two encodings should produce the same result as going through UTF-8 with PostgreSQL's own generated maps (or, where the direct table is believed to be authoritative, the generated maps and the direct table should at least agree with each other and with the documented encoding semantics). If maintaining per-pair tables is not considered worthwhile, routing single-byte <-> single-byte conversions through UTF-8 would remove the inconsistency. **Impact:** silent data corruption for `convert()`; on non-UTF8 databases the same code path serves automatic client_encoding conversion and `COPY ... ENCODING`, so imported/exported data can be silently corrupted (verified on a KOI8R database for `COPY FROM ... ENCODING 'WIN866'`: input byte 0xBD =3D U+255C was stored as 0xBD =3D U+256B). **Environment / determinism:** no special settings or extensions; results are deterministic across repeated runs; reproduced identically when the input comes from a table column instead of a literal, and under `PREPARE`/`EXECUTE` with `plan_cache_mode=3Dforce_generic_plan`, so this is not constant-folding related.