Received: from malur.postgresql.org ([2a02:16a8:dc51::56]) by arkaria.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA384:256) (Exim 4.89) (envelope-from ) id 1fL7e6-0005hF-8H for pgadmin-hackers@arkaria.postgresql.org; Tue, 22 May 2018 13:48:10 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.89) (envelope-from ) id 1fL7e5-0007Nj-7w for pgadmin-hackers@arkaria.postgresql.org; Tue, 22 May 2018 13:48:09 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA384:256) (Exim 4.89) (envelope-from ) id 1fL7c7-0004do-95 for pgadmin-hackers@lists.postgresql.org; Tue, 22 May 2018 13:46:07 +0000 Received: from mx0b-00296801.pphosted.com ([148.163.153.148]) by magus.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA384:256) (Exim 4.89) (envelope-from ) id 1fL7c0-00087c-9C for pgadmin-hackers@postgresql.org; Tue, 22 May 2018 13:46:05 +0000 Received: from pps.filterd (m0114585.ppops.net [127.0.0.1]) by mx0b-00296801.pphosted.com (8.16.0.22/8.16.0.22) with SMTP id w4MDjvlE005740 for ; Tue, 22 May 2018 13:45:57 GMT Received: from mail-io0-f200.google.com (mail-io0-f200.google.com [209.85.223.200]) by mx0b-00296801.pphosted.com with ESMTP id 2j2c34bdne-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Tue, 22 May 2018 13:45:57 +0000 Received: by mail-io0-f200.google.com with SMTP id w1-v6so15073433iod.1 for ; Tue, 22 May 2018 06:45:57 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=c+OwoGQE/VmY1uFAMszp5yFwlTlUXu8R7eDEIsWoFuo=; b=HkJ7pgOL6d8AV4hi8QhYotXJFoy1UqvL6obktxrFogtU6f/iiWsN4AP7AbTJvcMRe4 m0qpQLaFYI/cipIv2gmWVWQLl7bTBIgMPBg04/y6w3aFUlqrbhQuamIND//mCXkMcUA4 WfGXmx3iON/ZduYVy0s3HUFyX8RFWicOrE2Y21b8GbIozi6JmUiHhrOXGrlo6aKKGzWT e+THPMzAHJt9K8adAC8WqnEqRL0xK+BxN+jNlT2ZFf8l0ZUItTh9Jgyf3k+BKoQrgUZi 3wpkYz1EJDvFiwWV8RNsAJpt/pVSL/v6Fq/PgCk3/r0+eseW0Ypj9J/NUVgpAb830jYy bt0w== X-Gm-Message-State: ALKqPweLyKoizwC2JnMlWVxeFBnKQ6lcc6zL4v6APVMUxj9pjI7RYIeD ceyElG2Uzr36qWfQJTk6/GSn8jZNAUmyV1T5i6dGyT+ivLFenDHVZ6fGHoyjxcM2+ftbIHgZA4Y /RbKQEm9mrkUsksq8pcN7u4k6e0/OvNyzZPf4+EnbfQ3loRnpX7hY4NRpK+r8BNieCJMW X-Received: by 2002:a6b:6b02:: with SMTP id g2-v6mr26573804ioc.250.1526996756410; Tue, 22 May 2018 06:45:56 -0700 (PDT) X-Google-Smtp-Source: AB8JxZoSxhNisxwmNTicuABIMJ33UAa5Dskp524ApezZyib0gSeIXPNqKXsRDzmjzWHxv1hqXP2Sbbs6KXyAtR6ui60= X-Received: by 2002:a6b:6b02:: with SMTP id g2-v6mr26573782ioc.250.1526996756052; Tue, 22 May 2018 06:45:56 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Victoria Henry Date: Tue, 22 May 2018 09:45:44 -0400 Message-ID: Subject: Re: [pgAdmin4][RM#3289] Can't query SQL_ASCII database. To: Aditya Toshniwal Cc: Dave Page , pgadmin-hackers Content-Type: multipart/mixed; boundary="0000000000003806ab056ccba341" X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2018-05-22_03:,, signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=10 phishscore=0 bulkscore=0 spamscore=0 clxscore=1011 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1711220000 definitions=main-1805220161 List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Precedence: bulk --0000000000003806ab056ccba341 Content-Type: multipart/alternative; boundary="0000000000003806a6056ccba33f" --0000000000003806a6056ccba33f Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi Aditya, We made a minor change to make the patch so the python linter can pass. Attached is the change we made. Everything else looks good. Sincerely, Victoria & Anthony On Tue, May 22, 2018 at 4:46 AM Aditya Toshniwal < aditya.toshniwal@enterprisedb.com> wrote: > Hi, > > PFA updated patch. Linter issues are fixed ( we dont have any linter setu= p > for python :-( ) > Regarding test cases, they run successfully on my system and the reason i= t > failed for pivotal is timeout exception. I am sorry I can't help with tha= t. > > Traceback (most recent call last): > File > "/tmp/build/a453582b/pgadmin-repo/web/pgadmin/feature_tests/keyboard_shor= tcut_test.py", > line 52, in runTest > self._check_shortcuts() > File > "/tmp/build/a453582b/pgadmin-repo/web/pgadmin/feature_tests/keyboard_shor= tcut_test.py", > line 77, in _check_shortcuts > ") and contains(@class, 'open')]") > File > "/root/.pyenv/versions/pgadmin36/lib/python3.6/site-packages/selenium/web= driver/support/wait.py", > line 80, in until > raise TimeoutException(message, screen, stacktrace) > selenium.common.exceptions.TimeoutException: Message: > > Thanks and Regards, > Aditya Toshniwal > Software Engineer | EnterpriseDB Software Solutions | Pune > "Don't Complain about Heat, Plant a tree" > > On Tue, May 22, 2018 at 1:37 PM, Dave Page wrote: > >> Hi >> >> Pivotal's buildbot is showing problems with this patch: >> >> >> https://gpdb-dev.bosh.pivotalci.info/teams/pgadmin/pipelines/pgadmin-pat= ch/jobs/run-linter/builds/66 >> (linter failed) >> >> https://gpdb-dev.bosh.pivotalci.info/teams/pgadmin/pipelines/pgadmin-pat= ch/jobs/run-tests/builds/84 >> (tests failed) >> >> >> On Tue, May 22, 2018 at 7:05 AM, Aditya Toshniwal < >> aditya.toshniwal@enterprisedb.com> wrote: >> >>> Hi Hackers, >>> >>> PFA patch for RM#3289 where decode error was thrown on querying a >>> SQL_ASCII database table. Please note, this problem occurs only on wind= ows. >>> Sample insert - insert into test_tab values ('=C3=A9'); >>> >>> psycopg2 has a encodings dictionary where Postgres Database Encodings >>> are mapped to python equivalent. It uses 'ascii' decoder of python to >>> decode for SQL_ASCII encoding. If data has characters beyond the limit = of >>> ascii then it failed. The solution would be to use utf_8 decoder instea= d of >>> ascii. I tried setting the client_encoding using >>> set_client_encoding('UTF8') method of a psycopg2 connection but no luck >>> (also its not allowed for async connection). I also tried executing "SE= T >>> CLIENT_ENCODING=3D'UTF8'" but it didn't work too. >>> So, as in the patch, I had to set encodings dict value directly to >>> 'utf_8' and it seems to be working. Please note, the same is added to >>> psycopg3 milestones >>> https://github.com/psycopg/psycopg2/milestone/4 >>> >>> Also fixed a small glitch for sql editor connection status check. >>> >>> Kindly review. >>> >>> Thanks and Regards, >>> Aditya Toshniwal >>> Software Engineer | EnterpriseDB Software Solutions | Pune >>> "Don't Complain about Heat, Plant a tree" >>> >> >> >> >> -- >> Dave Page >> Blog: http://pgsnake.blogspot.com >> Twitter: @pgsnake >> >> EnterpriseDB UK: http://www.enterprisedb.com >> The Enterprise PostgreSQL Company >> > > --0000000000003806a6056ccba33f Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi Aditya,

We made a minor change to ma= ke the patch so the python linter can pass.=C2=A0 Attached is the change we= made.
Everything else looks good.

Since= rely,

Victoria & Anthony

On Tue, May 22, 2018 at 4:46 AM Adity= a Toshniwal <aditya= .toshniwal@enterprisedb.com> wrote:
Hi,

PFA updated patch. Linter i= ssues are fixed ( we dont have any linter setup for python :-( )
= Regarding test cases, they run successfully on my system and the reason it = failed for pivotal is timeout exception. I am sorry I can't help with t= hat.

Traceback (most recent call last):
=C2=A0 File = "/tmp/build/a453582b/pgadmin-repo/web/pgadmin/feature_tests/keyboard_s= hortcut_test.py", line 52, in runTest
=C2=A0 =C2=A0 self._check_sho= rtcuts()
=C2=A0 File "/tmp/build/a453582b/pgadmin-repo/web/pgadmin/= feature_tests/keyboard_shortcut_test.py", line 77, in _check_shortcuts=
=C2=A0 =C2=A0 ") and contains(@class, 'open')]")
= =C2=A0 File "/root/.pyenv/versions/pgadmin36/lib/python3.6/site-packag= es/selenium/webdriver/support/wait.py", line 80, in until
=C2=A0 = =C2=A0 raise TimeoutException(message, screen, stacktrace)
selenium.comm= on.exceptions.TimeoutException: Message:
<= br clear=3D"all">
Thanks a= nd Regards,
Aditya T= oshniwal
Software Engineer |=C2=A0EnterpriseDB Software Solutions |=C2=A0Pune
"Don't Complain about Heat, Plant a tree"<= /div>

On Tue, May 22, 2018 at 1:37 PM, Dave Page <= span dir=3D"ltr"><dpage@pgadmin.org> wrote:
=

On Tue, May 22, 2018 at 7:05 AM, Aditya Toshniwal <aditya.toshniwal@enterprisedb.com> wrote:
Hi Hackers,
PFA patch for RM#3289 where decode error was thrown on queryin= g a SQL_ASCII database table. Please note, this problem occurs only on wind= ows.
Sample insert -=C2=A0insert into test_tab values ('=C3= =A9');

psycopg2 has a encodings dictionary whe= re Postgres Database Encodings are mapped to python equivalent. It uses = 9;ascii' decoder of python to decode for SQL_ASCII encoding. If data ha= s characters beyond the limit of ascii then it failed. The solution would b= e to use utf_8 decoder instead of ascii. I tried setting the client_encodin= g using set_client_encoding('UTF8') method of a psycopg2 connection= but no luck (also its not allowed for async connection). I also tried exec= uting "SET CLIENT_ENCODING=3D'UTF8'" but it didn't wo= rk too.
So, as in the patch, I had to set encodings dict value di= rectly to 'utf_8' and it seems to be working. Please note, the same= is added to psycopg3 milestones

Also fixed a small glitch f= or sql editor connection status check.

Kindly revi= ew.

=
Thanks and Regards,
Aditya Toshniwal
Software Engineer |=C2=A0EnterpriseDB Software Sol= utions |=C2=A0Pune
"Don't Complain about Heat, Plant a tr= ee"



--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @p= gsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL = Company

--0000000000003806a6056ccba33f-- --0000000000003806ab056ccba341 Content-Type: application/octet-stream; name="RM3289_v2.patch" Content-Disposition: attachment; filename="RM3289_v2.patch" Content-Transfer-Encoding: base64 Content-ID: X-Attachment-Id: f_jhhqcxt90 ZGlmZiAtLWdpdCBhL3dlYi9wZ2FkbWluL3Rvb2xzL3NxbGVkaXRvci9fX2luaXRfXy5weSBiL3dl Yi9wZ2FkbWluL3Rvb2xzL3NxbGVkaXRvci9fX2luaXRfXy5weQppbmRleCBjNzI1MDVhNC4uOWY0 MWNmMGEgMTAwNjQ0Ci0tLSBhL3dlYi9wZ2FkbWluL3Rvb2xzL3NxbGVkaXRvci9fX2luaXRfXy5w eQorKysgYi93ZWIvcGdhZG1pbi90b29scy9zcWxlZGl0b3IvX19pbml0X18ucHkKQEAgLTE0NzQs NyArMTQ3NCw4IEBAIGRlZiBxdWVyeV90b29sX3N0YXR1cyh0cmFuc19pZCk6CiAgICAgICAgICAg ICBlcnJvcm1zZz1lcnJvcl9tc2cKICAgICAgICAgKQogCi0gICAgaWYgY29ubiBhbmQgdHJhbnNf b2JqIGFuZCBzZXNzaW9uX29iajoKKyAgICBpZiBjb25uIGFuZCB0cmFuc19vYmogYW5kIHNlc3Np b25fb2JqIFwKKyAgICAgICAgICAgIGFuZCBjb25uLnRyYW5zYWN0aW9uX3N0YXR1cygpIGlzIG5v dCBOb25lOgogICAgICAgICBzdGF0dXMgPSBjb25uLnRyYW5zYWN0aW9uX3N0YXR1cygpCiAgICAg ICAgIHJldHVybiBtYWtlX2pzb25fcmVzcG9uc2UoCiAgICAgICAgICAgICBkYXRhPXsKZGlmZiAt LWdpdCBhL3dlYi9wZ2FkbWluL3V0aWxzL2RyaXZlci9wc3ljb3BnMi9jb25uZWN0aW9uLnB5IGIv d2ViL3BnYWRtaW4vdXRpbHMvZHJpdmVyL3BzeWNvcGcyL2Nvbm5lY3Rpb24ucHkKaW5kZXggMzE1 NjMxYzAuLjhjYTMwYjM5IDEwMDY0NAotLS0gYS93ZWIvcGdhZG1pbi91dGlscy9kcml2ZXIvcHN5 Y29wZzIvY29ubmVjdGlvbi5weQorKysgYi93ZWIvcGdhZG1pbi91dGlscy9kcml2ZXIvcHN5Y29w ZzIvY29ubmVjdGlvbi5weQpAQCAtNDksNiArNDksMTAgQEAgZWxzZToKIAogXyA9IGdldHRleHQK IAorIyBVc2UgdXRmXzggZGVjb2RlciBpbnN0ZWFkIG9mIHBzeWNvcGcyIGRlZmF1bHQgYXNjaWkg Zm9yCisjIFNRTF9BU0NJSSBkYXRhYmFzZSB0byBhdm9pZCBkZWNvZGUgZXJyb3JzCitlbmNvZGlu Z3NbJ1NRTF9BU0NJSSddID0gJ3V0Zl84JworZW5jb2RpbmdzWydTUUxBU0NJSSddID0gJ3V0Zl84 JwogCiAjIFJlZ2lzdGVyIGdsb2JhbCB0eXBlIGNhc3RlciB3aGljaCB3aWxsIGJlIGFwcGxpY2Fi bGUgdG8gYWxsIGNvbm5lY3Rpb25zLgogcmVnaXN0ZXJfZ2xvYmFsX3R5cGVjYXN0ZXJzKCkK --0000000000003806ab056ccba341--