Received: from malur.postgresql.org ([2a02:16a8:dc51::56]) by arkaria.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA384:256) (Exim 4.89) (envelope-from ) id 1fONO7-0005qh-Ie for pgadmin-hackers@arkaria.postgresql.org; Thu, 31 May 2018 13:13:07 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.89) (envelope-from ) id 1fONO6-0001IV-EE for pgadmin-hackers@arkaria.postgresql.org; Thu, 31 May 2018 13:13:06 +0000 Received: from makus.postgresql.org ([2001:4800:1501:1::229]) by malur.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA384:256) (Exim 4.89) (envelope-from ) id 1fONO5-0001IL-Sz for pgadmin-hackers@lists.postgresql.org; Thu, 31 May 2018 13:13:06 +0000 Received: from mail-wm0-x244.google.com ([2a00:1450:400c:c09::244]) by makus.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1fONO1-0005XS-OB for pgadmin-hackers@postgresql.org; Thu, 31 May 2018 13:13:04 +0000 Received: by mail-wm0-x244.google.com with SMTP id a67-v6so53746347wmf.3 for ; Thu, 31 May 2018 06:13:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pgadmin-org.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=Qa4ZT3KwG425S4+JXz9taC1rnKaw9t4JSfwpZkA5l7s=; b=bmxeJzP5+4XcAw+tVKJCekQZRKIeYLGNVi50/0tiYuJAr2JAtECQacBRW871bh3hc4 eINTuClhqwxGpCcbd7oRJnNy0+TiBcY7bVLRYmu/BooavJ5a4DjNV1zMccI/WnHHNTzD ph5Y1wdOrUFAb+uwlFjxyYeWtWai/wOJRcDrykFvTPHdX/Mb9rsjdxcGGLpCkVd1FZBY Zuc3EqqDiPXbnlzJ9Btd2jos1iyXgveg2JXEhz6nfQkiwUfhYgkmVJ8sMUnLaocUqpVf VGgc9TSU4V8GO6AjlmFRnsrtAQ1cPgDnVXR3NLH6lO433a7Ljz4wR2GWywWX5B7y03nU 2fqw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=Qa4ZT3KwG425S4+JXz9taC1rnKaw9t4JSfwpZkA5l7s=; b=IhmH0z7DnzlVV5pPYZ8wGgq9fjrL20KtHaZVUJav8fDnjtyGlXPi2F7+W2rNWxap+Q KmIw0utkuHaeYP9qu4nRpY7xHq3i/bnLJyXMVxqBOjzoGTfTe3vAvBpNzEaE0oK6AbFj /OjpD4a9GspT4gkfJhfKFifuv5E/gzLW+njrig+K351kYAKoxMjQRkPnxnMvTcPIjiao uNDoTrS5Lwm555oKHDvI7DLAaIB/wVlmGt4BH/3YS5AuPdjPLlPZs4MpvmbtYwKPoxeZ xTlE+O9vK4vU/GVUChEWJMe10ryMCUYai/PkAlr9fAk++bB6XgmI763bs5BRsGMYQCBs lQhA== X-Gm-Message-State: ALKqPwfiQB57u6eLjk1cmSMW47LhrY2psjYNmyIe8UXu7ovDxgpDsoPl jh4YZu2SnbeettRgqGgKr/e0Z+Vjc8GAZWC1YhoRCw== X-Google-Smtp-Source: ADUXVKJflPVQBW1dEt+ucD3Ra46QIFzBoY8UBAegF4osAlHPlW8lYoUWKUp3CK5Z5nfe2FfY409haeWOaPPaVJmp6dE= X-Received: by 2002:a1c:b7d4:: with SMTP id h203-v6mr4211966wmf.161.1527772379969; Thu, 31 May 2018 06:12:59 -0700 (PDT) MIME-Version: 1.0 Received: by 2002:a1c:55c2:0:0:0:0:0 with HTTP; Thu, 31 May 2018 06:12:59 -0700 (PDT) In-Reply-To: References: From: Dave Page Date: Thu, 31 May 2018 09:12:59 -0400 Message-ID: Subject: Re: [pgAdmin4][RM#3289] Can't query SQL_ASCII database. To: Aditya Toshniwal Cc: Joao De Almeida Pereira , pgadmin-hackers Content-Type: multipart/alternative; boundary="00000000000001ad26056d803a37" List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Precedence: bulk --00000000000001ad26056d803a37 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi On Thu, May 31, 2018 at 1:20 AM, Aditya Toshniwal < aditya.toshniwal@enterprisedb.com> wrote: > Hi Victoria/Joao, > > On Thu, May 31, 2018 at 2:06 AM, Joao De Almeida Pereira < > jdealmeidapereira@pivotal.io> wrote: > >> Hello Aditya, >> >> It looks ok and it passes CI. >> >> We have some recommendations: >> - These look like 2 different changes so they should be in separated >> commits >> > > If you are talking of set client_encoding, then its not a bug. Its a > choice given to Postgres DB user to change the encoding of the characters= . > Postgres will translate characters from Server Encoding to Client Encodin= g, > and will throw error like mentioned previously. This link will help bette= r > - https://www.postgresql.org/docs/10/static/multibyte.html > The actual bug was, even after changing the client encoding to SQL_ASCII, > pgAdmin4 was not able to show the output as it was failing in encoding by > psycopg2. The patch is for resolving that. > > >> - Do we have test coverage for the bug that you are talking about? If no= t >> we should, to ensure this problem does not happen again in a future chan= ge. >> > > It is not possible adding test cases for encoding related stuff, as > Postgres support a lot many different types of encoding and conversions > (refer above link) > I was going to ask the same thing. Per https://www.postgresql.org/docs/10/static/multibyte.html#id-1.6.10.5.7, every characterset except SQL_ASCII can be converted to UTF-8, so we only need to test that UTF-8 and some other charactersets besides SQL_ASCII work, and then separately that SQL_ASCII with characters known not to be in UTF-8 work. > >> Thanks >> Victoria && Joao >> >> On Wed, May 30, 2018 at 3:06 AM Aditya Toshniwal < >> aditya.toshniwal@enterprisedb.com> wrote: >> >>> Hi Hackers, >>> >>> PFA updated patch after all the permutations, combinations for encoding >>> for SQL_ASCII database. Also fixed a small glitch for sql editor >>> connection status check. >>> >>> Please note, ERROR: invalid byte sequence for encoding "UTF8": 0xe9 0x7= 3 >>> is a Postgres DB error and not pgAdmin4 error. >>> >>> >>> >>> You need to change client_encoding to the appropriate. After changing >>> client_encoding using command - set client_encoding=3D'XYZ', it will gi= ve not >>> give error. >>> >>> >>> >>> Thanks and Regards, >>> Aditya Toshniwal >>> Software Engineer | EnterpriseDB Software Solutions | Pune >>> "Don't Complain about Heat, Plant a tree" >>> >>> On Wed, May 23, 2018 at 10:13 AM, Aditya Toshniwal < >>> aditya.toshniwal@enterprisedb.com> wrote: >>> >>>> Thank you Victoria, Anthony. >>>> >>>> Thanks and Regards, >>>> Aditya Toshniwal >>>> Software Engineer | EnterpriseDB Software Solutions | Pune >>>> "Don't Complain about Heat, Plant a tree" >>>> >>>> On Tue, May 22, 2018 at 7:15 PM, Victoria Henry >>>> wrote: >>>> >>>>> Hi Aditya, >>>>> >>>>> We made a minor change to make the patch so the python linter can >>>>> pass. Attached is the change we made. >>>>> Everything else looks good. >>>>> >>>>> Sincerely, >>>>> >>>>> Victoria & Anthony >>>>> >>>>> On Tue, May 22, 2018 at 4:46 AM Aditya Toshniwal < >>>>> aditya.toshniwal@enterprisedb.com> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> PFA updated patch. Linter issues are fixed ( we dont have any linter >>>>>> setup for python :-( ) >>>>>> Regarding test cases, they run successfully on my system and the >>>>>> reason it failed for pivotal is timeout exception. I am sorry I can'= t help >>>>>> with that. >>>>>> >>>>>> Traceback (most recent call last): >>>>>> File "/tmp/build/a453582b/pgadmin-repo/web/pgadmin/feature_tests/k= eyboard_shortcut_test.py", >>>>>> line 52, in runTest >>>>>> self._check_shortcuts() >>>>>> File "/tmp/build/a453582b/pgadmin-repo/web/pgadmin/feature_tests/k= eyboard_shortcut_test.py", >>>>>> line 77, in _check_shortcuts >>>>>> ") and contains(@class, 'open')]") >>>>>> File "/root/.pyenv/versions/pgadmin36/lib/python3.6/site-packages >>>>>> /selenium/webdriver/support/wait.py", line 80, in until >>>>>> raise TimeoutException(message, screen, stacktrace) >>>>>> selenium.common.exceptions.TimeoutException: Message: >>>>>> >>>>>> Thanks and Regards, >>>>>> Aditya Toshniwal >>>>>> Software Engineer | EnterpriseDB Software Solutions | Pune >>>>>> "Don't Complain about Heat, Plant a tree" >>>>>> >>>>>> On Tue, May 22, 2018 at 1:37 PM, Dave Page wrote= : >>>>>> >>>>>>> Hi >>>>>>> >>>>>>> Pivotal's buildbot is showing problems with this patch: >>>>>>> >>>>>>> https://gpdb-dev.bosh.pivotalci.info/teams/pgadmin/pipelines >>>>>>> /pgadmin-patch/jobs/run-linter/builds/66 (linter failed) >>>>>>> https://gpdb-dev.bosh.pivotalci.info/teams/pgadmin/pipelines >>>>>>> /pgadmin-patch/jobs/run-tests/builds/84 (tests failed) >>>>>>> >>>>>>> >>>>>>> On Tue, May 22, 2018 at 7:05 AM, Aditya Toshniwal < >>>>>>> aditya.toshniwal@enterprisedb.com> wrote: >>>>>>> >>>>>>>> Hi Hackers, >>>>>>>> >>>>>>>> PFA patch for RM#3289 where decode error was thrown on querying a >>>>>>>> SQL_ASCII database table. Please note, this problem occurs only on= windows. >>>>>>>> Sample insert - insert into test_tab values ('=C3=A9'); >>>>>>>> >>>>>>>> psycopg2 has a encodings dictionary where Postgres Database >>>>>>>> Encodings are mapped to python equivalent. It uses 'ascii' decoder= of >>>>>>>> python to decode for SQL_ASCII encoding. If data has characters be= yond the >>>>>>>> limit of ascii then it failed. The solution would be to use utf_8 = decoder >>>>>>>> instead of ascii. I tried setting the client_encoding using >>>>>>>> set_client_encoding('UTF8') method of a psycopg2 connection but no= luck >>>>>>>> (also its not allowed for async connection). I also tried executin= g "SET >>>>>>>> CLIENT_ENCODING=3D'UTF8'" but it didn't work too. >>>>>>>> So, as in the patch, I had to set encodings dict value directly to >>>>>>>> 'utf_8' and it seems to be working. Please note, the same is added= to >>>>>>>> psycopg3 milestones >>>>>>>> https://github.com/psycopg/psycopg2/milestone/4 >>>>>>>> >>>>>>>> Also fixed a small glitch for sql editor connection status check. >>>>>>>> >>>>>>>> Kindly review. >>>>>>>> >>>>>>>> Thanks and Regards, >>>>>>>> Aditya Toshniwal >>>>>>>> Software Engineer | EnterpriseDB Software Solutions | Pune >>>>>>>> "Don't Complain about Heat, Plant a tree" >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Dave Page >>>>>>> Blog: http://pgsnake.blogspot.com >>>>>>> Twitter: @pgsnake >>>>>>> >>>>>>> EnterpriseDB UK: http://www.enterprisedb.com >>>>>>> The Enterprise PostgreSQL Company >>>>>>> >>>>>> >>>>>> >>>> >>> > --=20 Dave Page Blog: http://pgsnake.blogspot.com Twitter: @pgsnake EnterpriseDB UK: http://www.enterprisedb.com The Enterprise PostgreSQL Company --00000000000001ad26056d803a37 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi

On Thu, May 31, 2018 at 1:20 AM, Aditya Toshniwal &= lt;a= ditya.toshniwal@enterprisedb.com> wrote:
Hi Victoria/Joao,
On Thu, May 31, 2018 at 2:06 AM, Joao De Almeida Pereira <jdealmeidapereira@pivotal.io> wrote:
Hello Aditya,

It looks ok and it passes CI.

We have some= recommendations:
- These look like 2 different changes so they s= hould be in separated commits
=C2=A0
If you are talking of set client_encoding= , then its not a bug. Its a choice given to Postgres DB user to change the = encoding of the characters. Postgres will translate characters from Server = Encoding to Client Encoding, and will throw error like mentioned previously= . This link will help better -=C2=A0https://ww= w.postgresql.org/docs/10/static/multibyte.html
The actual bug was, even after changing the client encodin= g to SQL_ASCII, pgAdmin4 was not able to show the output as it was failing = in encoding by psycopg2. The patch is for resolving that.
<= span class=3D"gmail-">
=C2=A0
- Do we have test coverage for the bug = that you are talking about? If not we should, to ensure this problem does n= ot happen again in a future change.

=
It is not possible adding test cases for encoding related stuff= , as Postgres support a lot many different types of encoding and conversion= s (refer above link)=C2=A0

I was going to ask the same thing. Per=C2=A0https://www.pos= tgresql.org/docs/10/static/multibyte.html#id-1.6.10.5.7, every characte= rset except SQL_ASCII can be converted to UTF-8, so we only need to test th= at UTF-8 and some other charactersets besides SQL_ASCII work, and then sepa= rately that SQL_ASCII with characters known not to be in UTF-8 work.
<= div>=C2=A0

= Thanks
Victoria && Joao
<= br>
On We= d, May 30, 2018 at 3:06 AM Aditya Toshniwal <aditya.toshniwal@enterprisedb.com> wrote:
Hi Hackers,

<= /div>
PFA updated patch after all the permutations, combinations for en= coding for SQL_ASCII database.=C2=A0 Also fixed a small glitch for sql editor connect= ion status check.

Please note,=C2=A0ERROR: invalid byte sequence fo= r encoding "UTF8": 0xe9 0x73 is a Postgres DB error and not pgAdm= in4 error.

<Image Deleted>
=

You need to change client_encodi= ng to the appropriate. After changing client_encoding using command - set c= lient_encoding=3D'XYZ', it will give not give error.

=
<Image Deleted><= /span>

Thanks and Regards,
Aditya Toshniwal
Software Engineer |=C2=A0EnterpriseDB S= oftware Solutions |=C2=A0Pune
"Don't Complain about Heat,= Plant a tree"

On Wed, May 23, 2018 at 10:13 AM, Aditya Tos= hniwal <aditya.toshniwal@enterprisedb.com> wrote:
Thank you Victoria, Anthony.
Thanks and Regards,
Aditya Toshniwal
Software Engineer |=C2=A0EnterpriseD= B Software Solutions |=C2=A0Pune=
"Don't Complain about He= at, Plant a tree"

On Tue, May 22, 2= 018 at 7:15 PM, Victoria Henry <vhenry@pivotal.io> wrote:
Hi Adit= ya,

We made a minor change to make the patch so the pyth= on linter can pass.=C2=A0 Attached is the change we made.
Everyth= ing else looks good.

Sincerely,

Victoria & Anthony

=
On Tue, May 22, 2018 at 4:46 AM Aditya Toshniwal <aditya.to= shniwal@enterprisedb.com> wrote:
Hi,

PFA up= dated patch. Linter issues are fixed ( we dont have any linter setup for py= thon :-( )
Regarding test cases, they run successfully on my syst= em and the reason it failed for pivotal is timeout exception. I am sorry I = can't help with that.

Traceback (most recent call l= ast):
=C2=A0 File "/tmp/build/a453582b/pgadmin-repo/web/pgadmi= n/feature_tests/keyboard_shortcut_test.py", line 52, in runTest=C2=A0 =C2=A0 self._check_shortcuts()
=C2=A0 File "/tmp/build/a45= 3582b/pgadmin-repo/web/pgadmin/feature_tests/keyboard_shortcut_te= st.py", line 77, in _check_shortcuts
=C2=A0 =C2=A0 ") and cont= ains(@class, 'open')]")
=C2=A0 File "/root/.pyenv/vers= ions/pgadmin36/lib/python3.6/site-packages/selenium/webdriver/sup= port/wait.py", line 80, in until
=C2=A0 =C2=A0 raise TimeoutEx= ception(message, screen, stacktrace)
selenium.common.exceptions.Tim= eoutException: Message:

=
Thanks and Regards,=
Aditya Toshniwal
Software = Engineer |=C2=A0EnterpriseDB Software Solutions |=C2=A0Pune
"= Don't Complain about Heat, Plant a tree"
=

On Tue, May 22, 2018 at 1:37 PM, Dave Page <= span dir=3D"ltr"><dpage@pgadmin.org> wrote:
=

On Tue, May 22, 2018 at 7:05 AM, Aditya= Toshniwal <aditya.toshniwal@enterprisedb.com>= wrote:
Hi Hackers,

PFA patch for RM#3289 w= here decode error was thrown on querying a SQL_ASCII database table. Please= note, this problem occurs only on windows.
Sample insert -=C2=A0= insert into test_tab values ('=C3=A9');

ps= ycopg2 has a encodings dictionary where Postgres Database Encodings are map= ped to python equivalent. It uses 'ascii' decoder of python to deco= de for SQL_ASCII encoding. If data has characters beyond the limit of ascii= then it failed. The solution would be to use utf_8 decoder instead of asci= i. I tried setting the client_encoding using set_client_encoding('UTF8&= #39;) method of a psycopg2 connection but no luck (also its not allowed for= async connection). I also tried executing "SET CLIENT_ENCODING=3D'= ;UTF8'" but it didn't work too.
So, as in the patch,= I had to set encodings dict value directly to 'utf_8' and it seems= to be working. Please note, the same is added to psycopg3 milestones
=
=
Also fixed a small glitch for sql editor connection status c= heck.

Kindly review.

<= div class=3D"gmail-m_-9019081956268395818m_-7389465867613098911m_-688114867= 3904556785m_-4282663695819898292m_-5961462812703942652m_-926911925651826607= m_-2482385932849901172m_-3775254183948713282gmail_signature">
Thanks and Regards,
Aditya Toshniwal
Software Engineer |=C2=A0EnterpriseDB Software= Solutions |=C2=A0Pune
"Don't Complain about Heat, Plant = a tree"



--
D= ave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB U= K: http://www.ent= erprisedb.com
The Enterprise PostgreSQL Company







--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake=

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Compan= y
--00000000000001ad26056d803a37--