Received: from malur.postgresql.org ([2a02:16a8:dc51::56]) by arkaria.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA384:256) (Exim 4.89) (envelope-from ) id 1fOG0p-0000ti-IG for pgadmin-hackers@arkaria.postgresql.org; Thu, 31 May 2018 05:20:35 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.89) (envelope-from ) id 1fOG0l-0000oe-DV for pgadmin-hackers@arkaria.postgresql.org; Thu, 31 May 2018 05:20:31 +0000 Received: from makus.postgresql.org ([2001:4800:1501:1::229]) by malur.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA384:256) (Exim 4.89) (envelope-from ) id 1fOG0l-0000oU-0P for pgadmin-hackers@lists.postgresql.org; Thu, 31 May 2018 05:20:31 +0000 Received: from mail-lf0-x244.google.com ([2a00:1450:4010:c07::244]) by makus.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1fOG0c-0003FU-Lw for pgadmin-hackers@postgresql.org; Thu, 31 May 2018 05:20:29 +0000 Received: by mail-lf0-x244.google.com with SMTP id r2-v6so7770213lff.4 for ; Wed, 30 May 2018 22:20:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=enterprisedb-com.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=h5hY+fLJtiwWeZIdhJDuOUqrs3uM7tqB1euu0t8x3q8=; b=iHZTd2kFNW9gkqUIs9FIJQ0JOZ5+itDwqbXVmhg6celAItDSjuF8ByM80gKXze89Iv uE0R/q4TJY938gSo+CluSOQvsenPW98VEFI8BTaPns4yQLQoks+vOudNA5DpaqwTKrRW EhpZPbeTZiUf3PwqLQWPqDHVfEGjEeGbkGAX3+1cNX1uMxToMI5eVK2Cr4ihBBRbSQfc 7A8tKKcmMqFcOwxM3a4heFbubt9xT4sYOeCEcMK+jijsWXoyu0UvPSyOHWUmK8ZQ+afp Thdu4uU/WRksNjTlLVqivy9BTCXdDCi1dIvn9i55N5GS17PVXytJ+waxDuS7h7Tc4uZQ Lh5A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=h5hY+fLJtiwWeZIdhJDuOUqrs3uM7tqB1euu0t8x3q8=; b=lH6GvVfJa+QrR5QhYl/YTYL+49JKggl8mDC1QaRARlfjRxwiF3MFLbXGJhmfSldVDR xOUxn0LYgLVGlagb3UFnp42wUEFL3A7ZlL87lEzlQiA1dIFq4KCqBVFm45qkK4WQ8FTL CTYICDJWhvgBCdmbq9y6tC0G0Xm+uAv+9/sRdANwC2IJLohfdx4E9ZvK3S6Nl3lTTlxs vh8Wm41TbmP9FOzkTUbFC8TJtX3laoAZra0enkBkU2CWky9Mt70E6h7/SzE1PuZ9uYx6 fYfo9mNy6X4EBlc+xXd54y/pUbt4M0IsRFdPySPRP6oULgs/eV5G03Da0RIO6G5SSTwZ ZVLA== X-Gm-Message-State: ALKqPwfKIf4TyVfbrCqHEgri1XMmZ6XlcLEMrWJkuW8/Z+9rE+XPIwH6 h+S600kNJmaEGys6qpfp+AIb0awAhv23UQvax9ac6Q== X-Google-Smtp-Source: ADUXVKKpcuvyBKLocl4Do+jJbRNr5ufl/IwEKQk1tIgS/RWBHO2XbC06v6vq5Ogf6PzeFowif5QQFNW2PGkFTwZ5MsQ= X-Received: by 2002:a19:5c4b:: with SMTP id q72-v6mr3179336lfb.128.1527744019792; Wed, 30 May 2018 22:20:19 -0700 (PDT) MIME-Version: 1.0 Received: by 2002:a2e:9e8a:0:0:0:0:0 with HTTP; Wed, 30 May 2018 22:20:19 -0700 (PDT) In-Reply-To: References: From: Aditya Toshniwal Date: Thu, 31 May 2018 10:50:19 +0530 Message-ID: Subject: Re: [pgAdmin4][RM#3289] Can't query SQL_ASCII database. To: Joao De Almeida Pereira Cc: pgadmin-hackers Content-Type: multipart/alternative; boundary="0000000000009b99ff056d799fcc" List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Precedence: bulk --0000000000009b99ff056d799fcc Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi Victoria/Joao, On Thu, May 31, 2018 at 2:06 AM, Joao De Almeida Pereira < jdealmeidapereira@pivotal.io> wrote: > Hello Aditya, > > It looks ok and it passes CI. > > We have some recommendations: > - These look like 2 different changes so they should be in separated > commits > If you are talking of set client_encoding, then its not a bug. Its a choice given to Postgres DB user to change the encoding of the characters. Postgres will translate characters from Server Encoding to Client Encoding, and will throw error like mentioned previously. This link will help better - https://www.postgresql.org/docs/10/static/multibyte.html The actual bug was, even after changing the client encoding to SQL_ASCII, pgAdmin4 was not able to show the output as it was failing in encoding by psycopg2. The patch is for resolving that. > - Do we have test coverage for the bug that you are talking about? If not > we should, to ensure this problem does not happen again in a future chang= e. > It is not possible adding test cases for encoding related stuff, as Postgres support a lot many different types of encoding and conversions (refer above link) > > Thanks > Victoria && Joao > > On Wed, May 30, 2018 at 3:06 AM Aditya Toshniwal enterprisedb.com> wrote: > >> Hi Hackers, >> >> PFA updated patch after all the permutations, combinations for encoding >> for SQL_ASCII database. Also fixed a small glitch for sql editor >> connection status check. >> >> Please note, ERROR: invalid byte sequence for encoding "UTF8": 0xe9 0x73 >> is a Postgres DB error and not pgAdmin4 error. >> >> >> >> You need to change client_encoding to the appropriate. After changing >> client_encoding using command - set client_encoding=3D'XYZ', it will giv= e not >> give error. >> >> >> >> Thanks and Regards, >> Aditya Toshniwal >> Software Engineer | EnterpriseDB Software Solutions | Pune >> "Don't Complain about Heat, Plant a tree" >> >> On Wed, May 23, 2018 at 10:13 AM, Aditya Toshniwal > enterprisedb.com> wrote: >> >>> Thank you Victoria, Anthony. >>> >>> Thanks and Regards, >>> Aditya Toshniwal >>> Software Engineer | EnterpriseDB Software Solutions | Pune >>> "Don't Complain about Heat, Plant a tree" >>> >>> On Tue, May 22, 2018 at 7:15 PM, Victoria Henry >>> wrote: >>> >>>> Hi Aditya, >>>> >>>> We made a minor change to make the patch so the python linter can >>>> pass. Attached is the change we made. >>>> Everything else looks good. >>>> >>>> Sincerely, >>>> >>>> Victoria & Anthony >>>> >>>> On Tue, May 22, 2018 at 4:46 AM Aditya Toshniwal >>> enterprisedb.com> wrote: >>>> >>>>> Hi, >>>>> >>>>> PFA updated patch. Linter issues are fixed ( we dont have any linter >>>>> setup for python :-( ) >>>>> Regarding test cases, they run successfully on my system and the >>>>> reason it failed for pivotal is timeout exception. I am sorry I can't= help >>>>> with that. >>>>> >>>>> Traceback (most recent call last): >>>>> File "/tmp/build/a453582b/pgadmin-repo/web/pgadmin/feature_ >>>>> tests/keyboard_shortcut_test.py", line 52, in runTest >>>>> self._check_shortcuts() >>>>> File "/tmp/build/a453582b/pgadmin-repo/web/pgadmin/feature_ >>>>> tests/keyboard_shortcut_test.py", line 77, in _check_shortcuts >>>>> ") and contains(@class, 'open')]") >>>>> File "/root/.pyenv/versions/pgadmin36/lib/python3.6/site- >>>>> packages/selenium/webdriver/support/wait.py", line 80, in until >>>>> raise TimeoutException(message, screen, stacktrace) >>>>> selenium.common.exceptions.TimeoutException: Message: >>>>> >>>>> Thanks and Regards, >>>>> Aditya Toshniwal >>>>> Software Engineer | EnterpriseDB Software Solutions | Pune >>>>> "Don't Complain about Heat, Plant a tree" >>>>> >>>>> On Tue, May 22, 2018 at 1:37 PM, Dave Page wrote: >>>>> >>>>>> Hi >>>>>> >>>>>> Pivotal's buildbot is showing problems with this patch: >>>>>> >>>>>> https://gpdb-dev.bosh.pivotalci.info/teams/pgadmin/ >>>>>> pipelines/pgadmin-patch/jobs/run-linter/builds/66 (linter failed) >>>>>> https://gpdb-dev.bosh.pivotalci.info/teams/pgadmin/ >>>>>> pipelines/pgadmin-patch/jobs/run-tests/builds/84 (tests failed) >>>>>> >>>>>> >>>>>> On Tue, May 22, 2018 at 7:05 AM, Aditya Toshniwal >>>>> enterprisedb.com> wrote: >>>>>> >>>>>>> Hi Hackers, >>>>>>> >>>>>>> PFA patch for RM#3289 where decode error was thrown on querying a >>>>>>> SQL_ASCII database table. Please note, this problem occurs only on = windows. >>>>>>> Sample insert - insert into test_tab values ('=C3=A9'); >>>>>>> >>>>>>> psycopg2 has a encodings dictionary where Postgres Database >>>>>>> Encodings are mapped to python equivalent. It uses 'ascii' decoder = of >>>>>>> python to decode for SQL_ASCII encoding. If data has characters bey= ond the >>>>>>> limit of ascii then it failed. The solution would be to use utf_8 d= ecoder >>>>>>> instead of ascii. I tried setting the client_encoding using >>>>>>> set_client_encoding('UTF8') method of a psycopg2 connection but no = luck >>>>>>> (also its not allowed for async connection). I also tried executing= "SET >>>>>>> CLIENT_ENCODING=3D'UTF8'" but it didn't work too. >>>>>>> So, as in the patch, I had to set encodings dict value directly to >>>>>>> 'utf_8' and it seems to be working. Please note, the same is added = to >>>>>>> psycopg3 milestones >>>>>>> https://github.com/psycopg/psycopg2/milestone/4 >>>>>>> >>>>>>> Also fixed a small glitch for sql editor connection status check. >>>>>>> >>>>>>> Kindly review. >>>>>>> >>>>>>> Thanks and Regards, >>>>>>> Aditya Toshniwal >>>>>>> Software Engineer | EnterpriseDB Software Solutions | Pune >>>>>>> "Don't Complain about Heat, Plant a tree" >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Dave Page >>>>>> Blog: http://pgsnake.blogspot.com >>>>>> Twitter: @pgsnake >>>>>> >>>>>> EnterpriseDB UK: http://www.enterprisedb.com >>>>>> The Enterprise PostgreSQL Company >>>>>> >>>>> >>>>> >>> >> --0000000000009b99ff056d799fcc Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi Victoria/Joao,

On Thu, May 31, 2018 at 2:06 AM, Joao De Almeid= a Pereira <jdealmeidapereira@pivotal.io> wrote:
Hello Aditya,

It looks ok and it passes CI.

We have some= recommendations:
- These look like 2 different changes so they s= hould be in separated commits
=C2=A0
If you are talking of set client_encoding, then = its not a bug. Its a choice given to Postgres DB user to change the encodin= g of the characters. Postgres will translate characters from Server Encodin= g to Client Encoding, and will throw error like mentioned previously. This = link will help better -=C2=A0https://www.postgresql.org/docs/10/= static/multibyte.html
The actual bu= g was, even after changing the client encoding to SQL_ASCII, pgAdmin4 was n= ot able to show the output as it was failing in encoding by psycopg2. The p= atch is for resolving that.
=C2=A0
- Do we have test= coverage for the bug that you are talking about? If not we should, to ensu= re this problem does not happen again in a future change.

It is not possible adding test cases for encodin= g related stuff, as Postgres support a lot many different types of encoding= and conversions (refer above link)=C2=A0

Thanks
Victoria && Joao

On Wed, May 30, 2018 at 3= :06 AM Aditya Toshniwal <aditya.toshniwal@enterprisedb.com> wrot= e:
Hi Hackers,

PFA updated patch after all the permutat= ions, combinations for encoding for SQL_ASCII database.=C2=A0 Also fixed a small glitch for sql editor connect= ion status check.

Please note,=C2=A0ERROR: invalid byte sequence fo= r encoding "UTF8": 0xe9 0x73 is a Postgres DB error and not pgAdm= in4 error.

<Image Deleted>
You need to change client_encoding to the appropriate. After c= hanging client_encoding using command - set client_encoding=3D'XYZ'= , it will give not give error.

<Image Deleted>
<= div class=3D"gmail_extra">
Thanks and Regards,
Aditya Toshniwal
Software Engineer |=C2= =A0EnterpriseDB Software Solutions |=C2=A0Pune
"Don't Com= plain about Heat, Plant a tree"
<= /div>

On Wed, May 23, 2018 at 10:13 AM, Aditya Tos= hniwal <aditya.toshniwal@enterprisedb.com> wrote:
Thank you Vic= toria, Anthony.

Thanks and Regards,
Aditya Toshniwal
Software Engineer |=C2=A0EnterpriseDB Software S= olutions |=C2=A0Pune
<= span style=3D"font-family:"trebuchet ms",sans-serif;font-size:x-s= mall">"Don't Complain about Heat, Plant a = tree"

On Tue, May 22, 2018 at 7:15 PM, Victoria Hen= ry <vhenry@pivotal.io> wrote:
Hi Aditya,

We made a minor change t= o make the patch so the python linter can pass.=C2=A0 Attached is the chang= e we made.
Everything else looks good.

S= incerely,

Victoria & Anthony

On Tue, May= 22, 2018 at 4:46 AM Aditya Toshniwal <aditya.toshniwal@enterprisedb.co= m> wrote:
H= i,

PFA updated patch. Linter issues are fixed ( we dont = have any linter setup for python :-( )
Regarding test cases, they= run successfully on my system and the reason it failed for pivotal is time= out exception. I am sorry I can't help with that.

T= raceback (most recent call last):
=C2=A0 File "/tmp/build/a453582b/= pgadmin-repo/web/pgadmin/feature_tests/keyboard_shortcut_test.py", line 52, in runTest
=C2=A0 =C2=A0 self._check_shortcuts()=C2=A0 File "/tmp/build/a453582b/pgadmin-repo/web/pgadmin/featur= e_tests/keyboard_shortcut_test.py", line 77, in _check_short= cuts
=C2=A0 =C2=A0 ") and contains(@class, 'open')]")<= br>=C2=A0 File "/root/.pyenv/versions/pgadmin36/lib/python3.6/sit= e-packages/selenium/webdriver/support/wait.py", line 80, in = until
=C2=A0 =C2=A0 raise TimeoutException(message, screen, stacktrace)<= br>selenium.common.exceptions.TimeoutException: Message:

Thanks and Regards,
Aditya Toshniwal
Software Engineer |=C2=A0EnterpriseDB S= oftware Solutions |=C2=A0Pune
"Don't Complain about Heat,= Plant a tree"

On Tue, May 22, 2018 at 1:37 PM, Dave Page <= span dir=3D"ltr"><dpage@pgadmin.org> wrote:
=
Hi

Pivotal's buildbot is showing pr= oblems with this patch:


On Tue, M= ay 22, 2018 at 7:05 AM, Aditya Toshniwal <aditya.toshniwal= @enterprisedb.com> wrote:
Hi Hackers,

PFA patch fo= r RM#3289 where decode error was thrown on querying a SQL_ASCII database ta= ble. Please note, this problem occurs only on windows.
Sample ins= ert -=C2=A0insert into test_tab values ('=C3=A9');

psycopg2 has a encodings dictionary where Postgres Database Encodi= ngs are mapped to python equivalent. It uses 'ascii' decoder of pyt= hon to decode for SQL_ASCII encoding. If data has characters beyond the lim= it of ascii then it failed. The solution would be to use utf_8 decoder inst= ead of ascii. I tried setting the client_encoding using set_client_encoding= ('UTF8') method of a psycopg2 connection but no luck (also its not = allowed for async connection). I also tried executing "SET CLIENT_ENCO= DING=3D'UTF8'" but it didn't work too.
So, as in= the patch, I had to set encodings dict value directly to 'utf_8' a= nd it seems to be working. Please note, the same is added to psycopg3 miles= tones

Also fixed a small glitch for sql editor connecti= on status check.

Kindly review.

Thanks and Regards,
Aditya Toshniwal
Software Engineer |=C2=A0EnterpriseDB Software Solutions |=C2= =A0Pune
"Don't Complain about Heat, Plant a tree"



-= -
Dave Page
Blog: http://pgsnake.blogs= pot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
= The Enterprise PostgreSQL Company




--0000000000009b99ff056d799fcc--