Subject: Proposal: tighten validation for legacy EUC encodings or document that accepted byte sequences may be unconvertible to UTF8

From: Zhongpu Chen <[email-masked]>
Date: Sat, 2 May 2026 10:31:12 +0800

See the related bug report
https://www.postgresql.org/message-id/CA%2B1gyqL7uiQhfLcYWpHNUKQgHjQc7sOPthSTiaxLDZzcrGFYSg%40mail.gmail.com

Currently PostgreSQL accepts structurally well-formed EUC_CN byte sequences
such as 0xA2A3 into text columns. The value round-trips when
client_encoding is EUC_CN, but fails when client_encoding is UTF8 because
euc_cn_to_utf8 has no mapping.

If this behavior is intentional for compatibility, the documentation should
explicitly say that validation for some legacy encodings is byte-structure
validation, not mapping-table validation.
If it is not intentional, stricter validation could reject unassigned byte
positions at input time.

-- 
Zhongpu Chen

---

From: Zhongpu Chen <[email-masked]>
Date: Sat, 2 May 2026 10:39:26 +0800

The issue is not specific to E'\\x..' literals. A normal COPY FROM data
file with ENCODING 'EUC_CN' can create text rows that later cannot be
retrieved with SELECT.

 This suggests that input validation for EUC_CN is only structural, while
the EUC_CN-to-UTF8 conversion table is stricter.

On Sat, May 2, 2026 at 10:31 AM Zhongpu Chen <[email-masked]> wrote:

> Currently PostgreSQL accepts structurally well-formed EUC_CN byte
> sequences such as 0xA2A3 into text columns. The value round-trips when
> If this behavior is intentional for compatibility, the documentation
> should explicitly say that validation for some legacy encodings is
> byte-structure validation, not mapping-table validation.

-- 
Zhongpu Chen

---

From: "David G. Johnston" <[email-masked]>
Date: Fri, 1 May 2026 20:28:31 -0700

> the EUC_CN-to-UTF8 conversion table is stricter.

I suspect a lack of desire to maintain and ensure that specific values are
verified; or accepting the runtime cost to do so.  It is indeed
structural.  This point should probably be documented better.  But it’s
hard to feel too bad if the input claims it is providing verifiable EUC_CN
data then proceeds to supply data that lacks meaning in reality.  We are
happy to just store and return your data to you - but it’s unreasonable to
ask for it to be converted.  It would be nice for the database to provide
an extra layer of protection, so I’m not against the idea.  Either
automatically or or at least providing a function that could, say, be
called in a trigger for opt-in.  But definitely feels like a problematic
benefit-to-cost proposition.

David J.

---

From: Zhongpu Chen <[email-masked]>
Date: Sat, 2 May 2026 12:49:00 +0800

Thanks for the clarification.

I agree that validation on every input may have runtime-cost concerns. But
this can be well-controlled. For example, MySQL adopts a finer checking for
EUC-CN (i.e., GB2312) in
https://github.com/mysql/mysql-server/blob/trunk/strings/ctype-gb2312.cc:

```

static int func_gb2312_uni_onechar(int code) {
  if ((code >= 0x2121) && (code <= 0x2658))
    return (tab_gb2312_uni0[code - 0x2121]);
  if ((code >= 0x2721) && (code <= 0x296F))
    return (tab_gb2312_uni1[code - 0x2721]);
  if ((code >= 0x3021) && (code <= 0x777E))
    return (tab_gb2312_uni2[code - 0x3021]);
  return (0);
}

```

where `code` is obtained by subtracting 0x8080. Of course, MySQL's checking
can also be enhanced.

Anyway, it is reasonable to note these details in the documentation.

-- 
Zhongpu Chen

---

From: Zhongpu Chen <[email-masked]>
Date: Wed, 6 May 2026 14:34:46 +0800

I run a benchmark to test the performance over a Chinese classic novel with
respect to various validation strategies:
https://github.com/SWUFE-DB-Group/NUAV/blob/main/encoding-validation/NUAV/src/gb2312.rs

The running log of `cargo bench -- gb2312`:

```
     Running benches/gb2312.rs (target/release/deps/gb2312-53d8e01b8e6785c8)
gb2312::is_gb2312_iconv time:   [2.5621 ms 2.5681 ms 2.5740 ms]
                        change: [-2.6404% -2.3284% -2.0023%] (p = 0.00 <
0.05)
                        Performance has improved.

gb2312::is_gb2312_icu   time:   [3.2427 ms 3.2589 ms 3.2771 ms]
                        change: [-1.5710% -1.0409% -0.4387%] (p = 0.00 <
0.05)
                        Change within noise threshold.
Found 5 outliers among 100 measurements (5.00%)
  3 (3.00%) high mild
  2 (2.00%) high severe

gb2312::is_gb2312_rs    time:   [2.8157 ms 2.8229 ms 2.8303 ms]
                        change: [-1.6985% -1.2165% -0.7501%] (p = 0.00 <
0.05)
                        Change within noise threshold.

Benchmarking gb2312::is_gb2312_range: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase
target time to 8.3s, enable flat sampling, or reduce sample count to 50.
gb2312::is_gb2312_range time:   [1.6237 ms 1.6294 ms 1.6351 ms]
                        change: [+3.8720% +4.2901% +4.6933%] (p = 0.00 <
0.05)
                        Performance has regressed.

gb2312::is_gb2312_lookup
                        time:   [488.12 µs 490.04 µs 491.97 µs]
                        change: [+0.9273% +2.2343% +3.2599%] (p = 0.00 <
0.05)
                        Change within noise threshold.
Found 1 outliers among 100 measurements (1.00%)
  1 (1.00%) low mild

gb2312::is_gb2312_simd  time:   [181.00 µs 181.77 µs 182.53 µs]
                        change: [-4.4563% -3.6971% -3.0260%] (p = 0.00 <
0.05)
                        Performance has improved.

gb2312:is_gb2312_ranges_pg
                        time:   [467.69 µs 469.27 µs 470.82 µs]

Benchmarking gb2312:is_gb2312_ranges_mysql: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase
target time to 6.4s, enable flat sampling, or reduce sample count to 60.
gb2312:is_gb2312_ranges_mysql
                        time:   [1.2611 ms 1.2667 ms 1.2724 ms]

```

As we can see, the PG-style validation does not bring much improvement.
Instead, it is slower than my strict-styles.

On Sat, May 2, 2026 at 12:49 PM Zhongpu Chen <[email-masked]> wrote:

> where `code` is obtained by subtracting 0x8080. Of course, MySQL's
> checking can also be enhanced.
>
>
>>>  This suggests that input validation for EUC_CN is only structural,
>>> while the EUC_CN-to-UTF8 conversion table is stricter.
>>>
>>
>> I suspect a lack of desire to maintain and ensure that specific values
>> are verified; or accepting the runtime cost to do so.  It is indeed

-- 
Zhongpu Chen

---

From: Peter Eisentraut <[email-masked]>
Date: Wed, 6 May 2026 09:32:23 +0200

On 02.05.26 04:31, Zhongpu Chen wrote:
> See the related bug report https://www.postgresql.org/message-id/ 
> CA%2B1gyqL7uiQhfLcYWpHNUKQgHjQc7sOPthSTiaxLDZzcrGFYSg%40mail.gmail.com 
> <https://www.postgresql.org/message-id/ 
> CA%2B1gyqL7uiQhfLcYWpHNUKQgHjQc7sOPthSTiaxLDZzcrGFYSg%40mail.gmail.com>
> 
> client_encoding is EUC_CN, but fails when client_encoding is UTF8 
> because euc_cn_to_utf8 has no mapping.
> 
> should explicitly say that validation for some legacy encodings is byte- 
> structure validation, not mapping-table validation.
> If it is not intentional, stricter validation could reject unassigned 
> byte positions at input time.

It is in general not necessarily required that all text in all non-UTF8 
encodings must be convertible to UTF8.

(This is also a result of history: These encodings were implemented in 
PostgreSQL before Unicode.)

That said, I can see how different behaviors might be desirable.

My first question would be, are these non-convertible byte sequences 
just characters that don't map to Unicode, or are they invalid within 
the definition of the EUC-* encodings themselves?  If the latter, then 
we should just reject them (modulo some backward compatibility), similar 
to how we reject certain Unicode code points that exist "structurally" 
but are not valid for one reason or another.

Alternatively, if these byte sequences are valid characters but they 
just didn't end up in Unicode for some reason, then rejecting them might 
break valid uses.

(I don't know much about EUC-* to be able to answer these.)

---

From: Zhongpu Chen <[email-masked]>
Date: Wed, 6 May 2026 17:15:18 +0800

I agree that not every valid character encoded in a legacy non-UTF8
encoding is necessarily required to be convertible to UTF8. But this
assumes that the byte sequence actually denotes a valid character in the
declared legacy encoding.

For the reported EUC-CN cases, this is exactly the point in question. These
byte sequences are structurally well-formed EUC-CN byte pairs, but they
fall into reserved or unassigned positions of the GB2312 code table. For
example, byte sequences with first byte 0xAA correspond to row 10 of
GB2312, which is reserved/unassigned. Therefore, these cases are not merely
valid legacy characters that happen to lack Unicode mappings. Rather, under
strict GB2312/EUC-CN semantics, they are not assigned to any character at
all, and thus should not be considered valid GB2312 characters.

So my concern is not that every legacy-encoded character must be
convertible to UTF8. The concern is that PostgreSQL's write-time validation
accepts a structural superset of EUC-CN byte pairs as text, while some of
these byte pairs are not valid assigned GB2312 characters and PostgreSQL's
own later conversion path cannot assign character semantics to them.

BTW, as noted in MySQL's implementation, a finer checker is possible.

-- 
Zhongpu Chen

---

From: Tatsuo Ishii <[email-masked]>
Date: Wed, 06 May 2026 21:19:07 +0900 (JST)

> It is in general not necessarily required that all text in all
> non-UTF8 encodings must be convertible to UTF8.
> 
> the definition of the EUC-* encodings themselves?

A strict answer is, the former. 0xA2A3 is 3 of lowercase forms of the
Roman numerals (iii), which is not defined in the original GB2312
(the character set of EUC_CN),

> If the latter, then
> we should just reject them (modulo some backward compatibility),
> similar to how we reject certain Unicode code points that exist
> "structurally" but are not valid for one reason or another.

After GB2312, GB18030 was defined. (It is claimed that GB18030 is a
super set of GB2312). In DB18030, lowercase forms of the Roman
numerals and other characters (e.g. Euro sign) were added.

So I suspect that a) those characters are sometimes used with EUC_CN
despite the fact that they are not valid GB2312 characters. b) some
EUC_CN users might have already written those characters into EUC_CN
databases. If so, tightening up the validation may break such that
uses. This is just my guess. Please correct me if I am wrong.

> just didn't end up in Unicode for some reason, then rejecting them
> might break valid uses.

That's not the case, at least for 0xA2A3. It seems UCS_ti_EUC_CN.pl
explicitly rejects characters that are not part of GB2312, including
0xA2A3, as the script is using GB18030 as a source data.

Regards,
--
Tatsuo Ishii
SRA OSS K.K.
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

---

From: Zhongpu Chen <[email-masked]>
Date: Sat, 9 May 2026 16:58:09 +0800

>  If so, tightening up the validation may break such that uses.

I agree. What about introducing an extra GUC which allows users to specify
verification logic? In fact, I have implemented this patch.

```
SHOW encoding_validation;
-- default behaviour
SET encoding_validation = 'native';
-- enforce Write to be fully compatible with Read
SET encoding_validation = 'read_compatible';
```

-- 
Zhongpu Chen

---

From: Zhongpu Chen <[email-masked]>
Date: Sun, 10 May 2026 10:28:57 +0800

My prototype implementation:
https://github.com/SWUFE-DB-Group/postgresql-encoding-validation and the
usage:
https://github.com/SWUFE-DB-Group/postgresql-encoding-validation/blob/main/DEV.md

-- 
Zhongpu Chen

---

From: Tatsuo Ishii <[email-masked]>
Date: Mon, 11 May 2026 10:40:13 +0900 (JST)

> SET encoding_validation = 'read_compatible';

-1 for using GUC. These settings may vary depending on the encoding.

Regards,

---

From: Tatsuo Ishii <[email-masked]>
Date: Mon, 11 May 2026 11:39:09 +0900 (JST)

[Add Cc: to pgsql-hackers]

From: Zhongpu Chen <[email-masked]>
Subject: Re: Proposal: tighten validation for legacy EUC encodings or document that accepted byte sequences may be unconvertible to UTF8
Date: Mon, 11 May 2026 09:56:20 +0800
Message-ID: <[email-masked]>

> I see. The settings may be used in a finer way. For example, `set
> euc-cn-encoding-valiation = 'read_compatible'`.

It will make pg_dumpall not working. Suppose there's a database
 populated with `set euc-cn-encoding-valiation = 'native'.

1. Dump the database cluster using pg_dumpall.
2. Create a new database cluster using initdb.
3. Set euc-cn-encoding-valiation = 'read_compatible' in the postgresql.conf.
4. Restore from the dump --- failure because of disallowed EUC_CN characters.

I think encoding properties (including character validation) should
belong to encoding itself, rather than GUC parameters. If you want to
have "strict" EUC_CN and "non-strict" EUC_CN at the same time, I think
the best way to implement it is, add new EUC_CN variant encoding.

Regards,