MIME-Version: 1.0
In-Reply-To: 
 <CA+1gyq+LF_91g_i0WXeKK6JGF8viaqaF213S-9Arq=SG=4GAaA@mail.gmail.com>
References: 
 <CA+1gyqJJJDhq=cc_D0ad59WH_OD2G_mN54xTru0KYoNaLkF48Q@mail.gmail.com>
 <CA+1gyq+LF_91g_i0WXeKK6JGF8viaqaF213S-9Arq=SG=4GAaA@mail.gmail.com>
From: "David G. Johnston" <david.g.johnston@gmail.com>
Date: Fri, 1 May 2026 20:28:31 -0700
Message-ID: 
 <CAKFQuwZuEZFYK9Arp_qFsoJ5o2EDDDCfsTwBYvoxzhBiXRJHQg@mail.gmail.com>
Subject: Re: Proposal: tighten validation for legacy EUC encodings or document
 that accepted byte sequences may be unconvertible to UTF8
To: Zhongpu Chen <chenloveit@gmail.com>
Cc: "pgsql-hackers@lists.postgresql.org" <pgsql-hackers@lists.postgresql.org>
Content-Type: multipart/alternative; boundary="000000000000bc74990650cd4a0c"
Archived-At: 
 <https://www.postgresql.org/message-id/CAKFQuwZuEZFYK9Arp_qFsoJ5o2EDDDCfsTwBYvoxzhBiXRJHQg%40mail.gmail.com>
Precedence: bulk

--000000000000bc74990650cd4a0c
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

On Friday, May 1, 2026, Zhongpu Chen <chenloveit@gmail.com> wrote:

> The issue is not specific to E'\\x..' literals. A normal COPY FROM data
> file with ENCODING 'EUC_CN' can create text rows that later cannot be
> retrieved with SELECT.
>
>  This suggests that input validation for EUC_CN is only structural, while
> the EUC_CN-to-UTF8 conversion table is stricter.
>

I suspect a lack of desire to maintain and ensure that specific values are
verified; or accepting the runtime cost to do so.  It is indeed
structural.  This point should probably be documented better.  But it=E2=80=
=99s
hard to feel too bad if the input claims it is providing verifiable EUC_CN
data then proceeds to supply data that lacks meaning in reality.  We are
happy to just store and return your data to you - but it=E2=80=99s unreason=
able to
ask for it to be converted.  It would be nice for the database to provide
an extra layer of protection, so I=E2=80=99m not against the idea.  Either
automatically or or at least providing a function that could, say, be
called in a trigger for opt-in.  But definitely feels like a problematic
benefit-to-cost proposition.

David J.

--000000000000bc74990650cd4a0c
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

On Friday, May 1, 2026, Zhongpu Chen &lt;<a href=3D"mailto:chenloveit@gmail=
.com">chenloveit@gmail.com</a>&gt; wrote:<br><blockquote class=3D"gmail_quo=
te" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"=
><div dir=3D"ltr"><div>The issue is not specific to E&#39;\\x..&#39; litera=
ls. A normal COPY FROM data file with ENCODING &#39;EUC_CN&#39; can create =
text rows that later cannot be retrieved with SELECT.</div><div><br></div><=
div>=C2=A0This suggests that input validation for EUC_CN is only structural=
, while the EUC_CN-to-UTF8 conversion table is stricter.</div></div></block=
quote><div><br></div><div>I suspect a lack of desire to maintain and ensure=
 that specific values are verified; or accepting the runtime cost to do so.=
=C2=A0 It is indeed structural.=C2=A0 This point should probably be documen=
ted better.=C2=A0 But it=E2=80=99s hard to feel too bad if the input claims=
 it is providing verifiable EUC_CN data then proceeds to supply data that l=
acks meaning in reality.=C2=A0 We are happy to just store and return your d=
ata to you - but it=E2=80=99s unreasonable to ask for it to be converted.=
=C2=A0 It would be nice for the database to provide an extra layer of prote=
ction, so I=E2=80=99m not against the idea.=C2=A0 Either automatically or o=
r at least providing a function that could, say, be called in a trigger for=
 opt-in.=C2=A0 But definitely feels like a problematic benefit-to-cost prop=
osition.</div><div><br></div><div>David J.</div><div><br></div>

--000000000000bc74990650cd4a0c--