MIME-Version: 1.0
References: <87a6shyenl.fsf@gmail.com>
In-Reply-To: <87a6shyenl.fsf@gmail.com>
From: Dave Page <dpage@pgadmin.org>
Date: Mon, 8 Feb 2021 09:25:27 +0000
Message-ID: 
 <CA+OCxoxsk-VJOJvw95V+pDN0G79T5-F1jAHB-THsPZd10f0KRg@mail.gmail.com>
Subject: Re: pgagent unicode support
To: Sergey Burladyan <eshkinkot@gmail.com>
Cc: pgadmin-hackers <pgadmin-hackers@postgresql.org>,
	Neel Patel <neel.patel@enterprisedb.com>,
 Ashesh Vashi <ashesh.vashi@enterprisedb.com>
Content-Type: multipart/alternative; boundary="000000000000c2de0005bacfc066"
Precedence: bulk

--000000000000c2de0005bacfc066
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

Hi

On Sat, Feb 6, 2021 at 5:00 AM Sergey Burladyan <eshkinkot@gmail.com> wrote=
:

> Currently pgagent doesn't handle unicode correctly.
>
> CharToWString function corrupt multibyte characters because it processes
> string one byte at a time:
>  148         std::string s =3D std::string(cstr);
>  149         std::wstring wsTmp(s.begin(), s.end());
>
> WStringToChar function does not take into account that there can be
> _multi_byte character on wcstombs output and create buffer with
> size =3D wcslen:
>  157         int wstr_length =3D wcslen(wchar_str);
>  158         char *dst =3D new char[wstr_length + 10];
>
> Also pgagent do not setup locale with setlocale(), without it all
> wcs/mbs functions cannot handle multibyte strings.
>
> For example:
>
> =3D=3D=3D step code =3D=3D=3D
> select '=D1=8D=D1=82=D0=BE =D0=BF=D1=80=D0=BE=D0=B2=D0=B5=D1=80=D0=BA=D0=
=B0 =D0=BA=D0=B8=D1=80=D0=B8=D0=BB=D0=BB=D0=B8=D1=86=D1=8B =D0=B2 =D1=82=D0=
=B5=D0=BB=D0=B5 =D0=B7=D0=B0=D0=BF=D1=80=D0=BE=D1=81=D0=B0 pgagent'
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>
> =3D=3D=3D postgres log =3D=3D=3D
> 2021-02-05 23:19:05 UTC [15600-1] postgres@postgres ERROR:  unterminated
> quoted string at or near "'" at character 8
> 2021-02-05 23:19:05 UTC [15600-2] postgres@postgres STATEMENT:  select '
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>
> Please see attached patch.
> I only test it on GNU/Linux and can't test it on Windows, sorry.
>

Thanks for the patch! Neel/Ashesh; can you take a look please? It looks OK
to me, but then I'm not overly familiar with multibyte string handling.
What, if anything, needs to be done on Windows?


--=20
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EDB: http://www.enterprisedb.com

--000000000000c2de0005bacfc066
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div>Hi</div><br><div class=3D"gmail_quote"><div dir=3D"lt=
r" class=3D"gmail_attr">On Sat, Feb 6, 2021 at 5:00 AM Sergey Burladyan &lt=
;<a href=3D"mailto:eshkinkot@gmail.com">eshkinkot@gmail.com</a>&gt; wrote:<=
br></div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8e=
x;border-left:1px solid rgb(204,204,204);padding-left:1ex">Currently pgagen=
t doesn&#39;t handle unicode correctly.<br>
<br>
CharToWString function corrupt multibyte characters because it processes<br=
>
string one byte at a time:<br>
=C2=A0148=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0std::string s =3D std::string(cs=
tr);<br>
=C2=A0149=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0std::wstring wsTmp(s.begin(), s.=
end());<br>
<br>
WStringToChar function does not take into account that there can be<br>
_multi_byte character on wcstombs output and create buffer with<br>
size =3D wcslen:<br>
=C2=A0157=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0int wstr_length =3D wcslen(wchar=
_str);<br>
=C2=A0158=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0char *dst =3D new char[wstr_leng=
th + 10];<br>
<br>
Also pgagent do not setup locale with setlocale(), without it all<br>
wcs/mbs functions cannot handle multibyte strings.<br>
<br>
For example:<br>
<br>
=3D=3D=3D step code =3D=3D=3D<br>
select &#39;=D1=8D=D1=82=D0=BE =D0=BF=D1=80=D0=BE=D0=B2=D0=B5=D1=80=D0=BA=
=D0=B0 =D0=BA=D0=B8=D1=80=D0=B8=D0=BB=D0=BB=D0=B8=D1=86=D1=8B =D0=B2 =D1=82=
=D0=B5=D0=BB=D0=B5 =D0=B7=D0=B0=D0=BF=D1=80=D0=BE=D1=81=D0=B0 pgagent&#39;<=
br>
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D<br>
<br>
=3D=3D=3D postgres log =3D=3D=3D<br>
2021-02-05 23:19:05 UTC [15600-1] postgres@postgres ERROR:=C2=A0 unterminat=
ed quoted string at or near &quot;&#39;&quot; at character 8<br>
2021-02-05 23:19:05 UTC [15600-2] postgres@postgres STATEMENT:=C2=A0 select=
 &#39;<br>
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D<br>
<br>
Please see attached patch.<br>
I only test it on GNU/Linux and can&#39;t test it on Windows, sorry.<br></b=
lockquote><div><br></div><div>Thanks for the patch! Neel/Ashesh; can you ta=
ke a look please? It looks OK to me, but then I&#39;m not overly familiar w=
ith multibyte string handling. What, if anything, needs to be done on Windo=
ws?</div><div>=C2=A0</div></div><div><br></div>-- <br><div dir=3D"ltr" clas=
s=3D"gmail_signature"><div dir=3D"ltr">Dave Page<br>Blog: <a href=3D"http:/=
/pgsnake.blogspot.com" target=3D"_blank">http://pgsnake.blogspot.com</a><br=
>Twitter: @pgsnake<br><br>EDB: <a href=3D"http://www.enterprisedb.com" targ=
et=3D"_blank">http://www.enterprisedb.com</a><br><br></div></div></div>

--000000000000c2de0005bacfc066--