MIME-Version: 1.0
References: 
 <CA+1gyqJJJDhq=cc_D0ad59WH_OD2G_mN54xTru0KYoNaLkF48Q@mail.gmail.com>
 <30e628b4-03cd-43eb-9ea4-d211aaddcaf5@eisentraut.org>
In-Reply-To: <30e628b4-03cd-43eb-9ea4-d211aaddcaf5@eisentraut.org>
From: Zhongpu Chen <chenloveit@gmail.com>
Date: Wed, 6 May 2026 17:15:18 +0800
Message-ID: 
 <CA+1gyqK5FuJUG6rDRaYXZBcPd=Wn2fuy9Qhp5X_DT=J3a2HMAA@mail.gmail.com>
Subject: Re: Proposal: tighten validation for legacy EUC encodings or document
 that accepted byte sequences may be unconvertible to UTF8
To: Peter Eisentraut <peter@eisentraut.org>
Cc: pgsql-hackers@lists.postgresql.org
Content-Type: multipart/alternative; boundary="000000000000e4b16f0651229a8d"
Archived-At: 
 <https://www.postgresql.org/message-id/CA%2B1gyqK5FuJUG6rDRaYXZBcPd%3DWn2fuy9Qhp5X_DT%3DJ3a2HMAA%40mail.gmail.com>
Precedence: bulk

--000000000000e4b16f0651229a8d
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

I agree that not every valid character encoded in a legacy non-UTF8
encoding is necessarily required to be convertible to UTF8. But this
assumes that the byte sequence actually denotes a valid character in the
declared legacy encoding.

For the reported EUC-CN cases, this is exactly the point in question. These
byte sequences are structurally well-formed EUC-CN byte pairs, but they
fall into reserved or unassigned positions of the GB2312 code table. For
example, byte sequences with first byte 0xAA correspond to row 10 of
GB2312, which is reserved/unassigned. Therefore, these cases are not merely
valid legacy characters that happen to lack Unicode mappings. Rather, under
strict GB2312/EUC-CN semantics, they are not assigned to any character at
all, and thus should not be considered valid GB2312 characters.

So my concern is not that every legacy-encoded character must be
convertible to UTF8. The concern is that PostgreSQL's write-time validation
accepts a structural superset of EUC-CN byte pairs as text, while some of
these byte pairs are not valid assigned GB2312 characters and PostgreSQL's
own later conversion path cannot assign character semantics to them.

BTW, as noted in MySQL's implementation, a finer checker is possible.


On Wed, May 6, 2026 at 3:32=E2=80=AFPM Peter Eisentraut <peter@eisentraut.o=
rg>
wrote:

> On 02.05.26 04:31, Zhongpu Chen wrote:
> > See the related bug report https://www.postgresql.org/message-id/
> > CA%2B1gyqL7uiQhfLcYWpHNUKQgHjQc7sOPthSTiaxLDZzcrGFYSg%40mail.gmail.com
> > <https://www.postgresql.org/message-id/
> > CA%2B1gyqL7uiQhfLcYWpHNUKQgHjQc7sOPthSTiaxLDZzcrGFYSg%40mail.gmail.com>
> >
> > Currently PostgreSQL accepts structurally well-formed EUC_CN byte
> > sequences such as 0xA2A3 into text columns. The value round-trips when
> > client_encoding is EUC_CN, but fails when client_encoding is UTF8
> > because euc_cn_to_utf8 has no mapping.
> >
> > If this behavior is intentional for compatibility, the documentation
> > should explicitly say that validation for some legacy encodings is byte=
-
> > structure validation, not mapping-table validation.
> > If it is not intentional, stricter validation could reject unassigned
> > byte positions at input time.
>
> It is in general not necessarily required that all text in all non-UTF8
> encodings must be convertible to UTF8.
>
> (This is also a result of history: These encodings were implemented in
> PostgreSQL before Unicode.)
>
> That said, I can see how different behaviors might be desirable.
>
> My first question would be, are these non-convertible byte sequences
> just characters that don't map to Unicode, or are they invalid within
> the definition of the EUC-* encodings themselves?  If the latter, then
> we should just reject them (modulo some backward compatibility), similar
> to how we reject certain Unicode code points that exist "structurally"
> but are not valid for one reason or another.
>
> Alternatively, if these byte sequences are valid characters but they
> just didn't end up in Unicode for some reason, then rejecting them might
> break valid uses.
>
> (I don't know much about EUC-* to be able to answer these.)
>
>

--=20
Zhongpu Chen

--000000000000e4b16f0651229a8d
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div>I agree that not every valid character encoded in a l=
egacy non-UTF8 encoding is necessarily required to be convertible to UTF8. =
But this assumes that the byte sequence actually denotes a valid character =
in the declared legacy encoding.<br><br>For the reported EUC-CN cases, this=
 is exactly the point in question. These byte sequences are structurally we=
ll-formed EUC-CN byte pairs, but they fall into reserved or unassigned posi=
tions of the GB2312 code table. For example, byte sequences with first byte=
 0xAA correspond to row 10 of GB2312, which is reserved/unassigned. Therefo=
re, these cases are not merely valid legacy characters that happen to lack =
Unicode mappings. Rather, under strict GB2312/EUC-CN semantics, they are no=
t assigned to any character at all, and thus should not be considered valid=
 GB2312 characters.<br><br>So my concern is not that every legacy-encoded c=
haracter must be convertible to UTF8. The concern is that PostgreSQL&#39;s =
write-time validation accepts a structural superset of EUC-CN byte pairs as=
 text, while some of these byte pairs are not valid assigned GB2312 charact=
ers and PostgreSQL&#39;s own later conversion path cannot assign character =
semantics to them.<br><br></div>BTW, as noted in MySQL&#39;s implementation=
, a finer checker is possible.<div><br></div></div><br><div class=3D"gmail_=
quote gmail_quote_container"><div dir=3D"ltr" class=3D"gmail_attr">On Wed, =
May 6, 2026 at 3:32=E2=80=AFPM Peter Eisentraut &lt;<a href=3D"mailto:peter=
@eisentraut.org">peter@eisentraut.org</a>&gt; wrote:<br></div><blockquote c=
lass=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px soli=
d rgb(204,204,204);padding-left:1ex">On 02.05.26 04:31, Zhongpu Chen wrote:=
<br>
&gt; See the related bug report <a href=3D"https://www.postgresql.org/messa=
ge-id/" rel=3D"noreferrer" target=3D"_blank">https://www.postgresql.org/mes=
sage-id/</a> <br>
&gt; CA%2B1gyqL7uiQhfLcYWpHNUKQgHjQc7sOPthSTiaxLDZzcrGFYSg%<a href=3D"http:=
//40mail.gmail.com" rel=3D"noreferrer" target=3D"_blank">40mail.gmail.com</=
a> <br>
&gt; &lt;<a href=3D"https://www.postgresql.org/message-id/" rel=3D"noreferr=
er" target=3D"_blank">https://www.postgresql.org/message-id/</a> <br>
&gt; CA%2B1gyqL7uiQhfLcYWpHNUKQgHjQc7sOPthSTiaxLDZzcrGFYSg%<a href=3D"http:=
//40mail.gmail.com" rel=3D"noreferrer" target=3D"_blank">40mail.gmail.com</=
a>&gt;<br>
&gt; <br>
&gt; Currently PostgreSQL accepts structurally well-formed EUC_CN byte <br>
&gt; sequences such as 0xA2A3 into text columns. The value round-trips when=
 <br>
&gt; client_encoding is EUC_CN, but fails when client_encoding is UTF8 <br>
&gt; because euc_cn_to_utf8 has no mapping.<br>
&gt; <br>
&gt; If this behavior is intentional for compatibility, the documentation <=
br>
&gt; should explicitly say that validation for some legacy encodings is byt=
e- <br>
&gt; structure validation, not mapping-table validation.<br>
&gt; If it is not intentional, stricter validation could reject unassigned =
<br>
&gt; byte positions at input time.<br>
<br>
It is in general not necessarily required that all text in all non-UTF8 <br=
>
encodings must be convertible to UTF8.<br>
<br>
(This is also a result of history: These encodings were implemented in <br>
PostgreSQL before Unicode.)<br>
<br>
That said, I can see how different behaviors might be desirable.<br>
<br>
My first question would be, are these non-convertible byte sequences <br>
just characters that don&#39;t map to Unicode, or are they invalid within <=
br>
the definition of the EUC-* encodings themselves?=C2=A0 If the latter, then=
 <br>
we should just reject them (modulo some backward compatibility), similar <b=
r>
to how we reject certain Unicode code points that exist &quot;structurally&=
quot; <br>
but are not valid for one reason or another.<br>
<br>
Alternatively, if these byte sequences are valid characters but they <br>
just didn&#39;t end up in Unicode for some reason, then rejecting them migh=
t <br>
break valid uses.<br>
<br>
(I don&#39;t know much about EUC-* to be able to answer these.)<br>
<br>
</blockquote></div><div><br clear=3D"all"></div><br><span class=3D"gmail_si=
gnature_prefix">-- </span><br><div dir=3D"ltr" class=3D"gmail_signature"><d=
iv dir=3D"ltr">Zhongpu Chen</div></div>

--000000000000e4b16f0651229a8d--