Received: from malur.postgresql.org ([2a02:16a8:dc51::56]) by arkaria.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA384:256) (Exim 4.89) (envelope-from ) id 1fQsIV-0007Tc-EX for pgadmin-hackers@arkaria.postgresql.org; Thu, 07 Jun 2018 10:37:40 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.89) (envelope-from ) id 1fQsIT-00059o-Rt for pgadmin-hackers@arkaria.postgresql.org; Thu, 07 Jun 2018 10:37:37 +0000 Received: from makus.postgresql.org ([2001:4800:1501:1::229]) by malur.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA384:256) (Exim 4.89) (envelope-from ) id 1fQsIT-00059e-1V for pgadmin-hackers@lists.postgresql.org; Thu, 07 Jun 2018 10:37:37 +0000 Received: from mail-wr0-x242.google.com ([2a00:1450:400c:c0c::242]) by makus.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1fQsIN-0007o3-PC for pgadmin-hackers@postgresql.org; Thu, 07 Jun 2018 10:37:35 +0000 Received: by mail-wr0-x242.google.com with SMTP id e18-v6so1286760wrs.5 for ; Thu, 07 Jun 2018 03:37:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pgadmin-org.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=MebqB4k53IQ7VxXAWTiXPpP8azB7w46Foyu/bRSUD+g=; b=py4pfSc/lxy5/Gj4dkKGxUDZbGBlMlo/zQlE6rTUV8f97QQ6VWX2bTdTvNWSwJNjjO 80pkmiOX7FJKuhTQQzPR6prHxoEdbxtiV+rFooa9BNMUGFqJjNXHbsG1gmjt4GMJKDht UaOx4GrnEo5qY3KqCQaMkOPrN12CxSwAnM9eJOiNjPbHiGv9pSl+krXsNKPHaFhEwGAG ViF5Pr3aUWKWOK97vNQHIHlyuxaeFg9eKp5Rw3dbw2vcU93edic3CSFWqmW/Odx2Ln5b JqdgDzQFcpe7TWnuPuyv85OBKQyrbiKgfpuZk5/GZnT+0Xe0kblEx+xlA35yaU1CuAX9 pzzA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=MebqB4k53IQ7VxXAWTiXPpP8azB7w46Foyu/bRSUD+g=; b=gC64qpWRKN79NkFfYG7OrqHv5kK+l0+aRPymdKW7vAreoYEtzjm1kVa5TITK0MI8zK gxK+mxIbXskbtDmTmWzgWYowTa0UYR1Bwg9+5v2MKjFgWDr1Y33Vam8b0HCbkGXSiWx7 vKDQQTfxuovOcv92ErJSFBZhoEvxa+HvYAWeVH26EUGy+uiYrsjI1VgW6HOd7DO8YAhH VxeNiJ5uJzfUx8dWTBy08wjdnEXW+chKC6jkErtQFhVJ6Qz5dai1fXG311wnsNjeSrus IKT5dlQub90IiQeMcvqN432NjvBJ53+oTP7q8ko4o6n4QDWAs3NVloqHtxOLU/GxIXW8 jGpg== X-Gm-Message-State: APt69E0T4L2+dYQHLKxYAMu2uetyOeNV7JMj57zNA9i17INV5YVlVy/y YvR+yYS5XsANq7U7kQw4h2J/h5YWcbcTI4g1P3WC1w== X-Google-Smtp-Source: ADUXVKIHzbAPI8qNtgCj3RGRyC4D8Qs8GzTY6cFuECj/LglgVOEqBKj172M4uVkxc3r2ylKpC0XP5umu/pXx6byZwTU= X-Received: by 2002:adf:f090:: with SMTP id n16-v6mr1294312wro.49.1528367849833; Thu, 07 Jun 2018 03:37:29 -0700 (PDT) MIME-Version: 1.0 Received: by 2002:a1c:2907:0:0:0:0:0 with HTTP; Thu, 7 Jun 2018 03:37:28 -0700 (PDT) In-Reply-To: References: From: Dave Page Date: Thu, 7 Jun 2018 11:37:28 +0100 Message-ID: Subject: Re: [pgAdmin4][RM#3289] Can't query SQL_ASCII database. To: Aditya Toshniwal Cc: Joao De Almeida Pereira , pgadmin-hackers Content-Type: multipart/alternative; boundary="000000000000c69f02056e0adeb0" List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Precedence: bulk --000000000000c69f02056e0adeb0 Content-Type: text/plain; charset="UTF-8" Hi On Wed, Jun 6, 2018 at 2:02 PM, Aditya Toshniwal wrote: > Hi Hackers, > > PFA updated patch as the previous one was not working as expected. I have > tried to make it similar to that of pgAdmin3 and you do not need to change > client_encoding as it is set now based on server encoding. It works fine > with "view data" also. > - In connection.py, at ~409, shouldn't we set the client_encoding to SQL_ASCII? Otherwise it could be overridden with something unexpected if the client has PGCLIENTENCODING set for example. - With or without that change, I get the following test failure on macOS with Python 2.7.10: ====================================================================== ERROR: runTest (pgadmin.tools.sqleditor.tests.test_encoding_charset.TestEncodingCharset) With Encoding SQL_ASCII ---------------------------------------------------------------------- Traceback (most recent call last): File "/Users/dpage/git/pgadmin4/web/pgadmin/tools/sqleditor/tests/test_encoding_charset.py", line 86, in runTest response = self.tester.get(url) File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/werkzeug/test.py", line 830, in get return self.open(*args, **kw) File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/testing.py", line 127, in open follow_redirects=follow_redirects) File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/werkzeug/test.py", line 803, in open response = self.run_wsgi_app(environ, buffered=buffered) File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/werkzeug/test.py", line 716, in run_wsgi_app rv = run_wsgi_app(self.application, environ, buffered=buffered) File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/werkzeug/test.py", line 923, in run_wsgi_app app_rv = app(environ, start_response) File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py", line 1985, in wsgi_app response = self.handle_exception(e) File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py", line 1540, in handle_exception reraise(exc_type, exc_value, tb) File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py", line 1982, in wsgi_app response = self.full_dispatch_request() File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py", line 1614, in full_dispatch_request rv = self.handle_user_exception(e) File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py", line 1517, in handle_user_exception reraise(exc_type, exc_value, tb) File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py", line 1612, in full_dispatch_request rv = self.dispatch_request() File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py", line 1598, in dispatch_request return self.view_functions[rule.endpoint](**req.view_args) File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask_login.py", line 792, in decorated_view return func(*args, **kwargs) File "/Users/dpage/git/pgadmin4/web/pgadmin/tools/sqleditor/__init__.py", line 576, in poll 'oids': oids File "/Users/dpage/git/pgadmin4/web/pgadmin/utils/ajax.py", line 61, in make_json_response separators=(',', ':')), File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/simplejson/__init__.py", line 399, in dumps **kw).encode(obj) File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/simplejson/encoder.py", line 291, in encode chunks = self.iterencode(o, _one_shot=True) File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/simplejson/encoder.py", line 373, in iterencode return _iterencode(o, 0) UnicodeDecodeError: 'utf8' codec can't decode byte 0xad in position 0: invalid start byte ---------------------------------------------------------------------- Ran 317 tests in 30.692s FAILED (errors=1, skipped=21) > The only problem is, I cannot find equivalent codec for wxConvLibc in > python. The closest one I could find is raw_unicode_escape. So, in a > SQL_ASCII database, non ASCII characters may differ in pgAdmin4 and > pgAdmin3, but it will display results. > Yeah, I think that's fine. For the small number of people with SQL_ASCII databases, seeing escaped characters is better than nothing. > > > Dave, > You need to add "E" before the string to be inserted, otherwise \x will be > considered as a plain string. > INSERT INTO sql_ascii (data) VALUES (E'[Invalid UTF-8] Blob: > \xf4\xa5\xa3\xa5'); > Yeah, sorry - I copied the wrong version of the query :-( > > > Kindly review. > > Thanks and Regards, > Aditya Toshniwal > Software Engineer | EnterpriseDB Software Solutions | Pune > "Don't Complain about Heat, Plant a tree" > > On Tue, Jun 5, 2018 at 6:42 PM, Dave Page wrote: > >> Hi >> >> On Tue, Jun 5, 2018 at 2:03 PM, Aditya Toshniwal < >> aditya.toshniwal@enterprisedb.com> wrote: >> >>> Hi >>> >>> On Tue, Jun 5, 2018 at 6:25 PM, Dave Page wrote: >>> >>>> >>>> >>>> On Tue, Jun 5, 2018 at 1:49 PM, Aditya Toshniwal < >>>> aditya.toshniwal@enterprisedb.com> wrote: >>>> >>>>> Hi Dave, >>>>> >>>>> The problem of SQL ASCII is solved with the patch, and not related to >>>>> setting the client encoding of the sql window. >>>>> >>>> >>>> No it's not. It doesn't work for me as I said (and showed the example >>>> of). >>>> >>> >>> After setting the client_encoding to SQL_ASCII you got the output. >>> Previously, it used to fail in the back end itself because python encoding >>> failure. That is fixed. >>> The error ERROR: invalid byte sequence for encoding "UTF8": 0x80 is >>> thrown by postgres and not python or pgAdmin4. You will get the same error >>> even if you >>> connect from psql. >>> >> >> Sure - but that is not a fix. You have no way of running the SET command >> if you're using "view data" - and in the query tool, users just expect it >> to work (as it did in pgAdmin 3). >> >> >>> >>>> >>>>> I can see there is no SET call in pgAdmin3 for client_encoding. I can >>>>> remove the SET client_encoding='UNICODE'; that will solve the >>>>> problem. But, can you please let me know why that was added. >>>>> >>>> >>>> There is, but it's inside an API call (PQsetClientEncoding): >>>> >>>> 300 >>>> >>>> wxLogInfo(wxT("Setting client_encoding to '%s'") >>>> , encoding.c_str()); >>>> 301 >>>> >>>> if (PQsetClientEncoding(conn, encoding.ToAscii())) >>>> 302 >>>> >>>> { >>>> 303 >>>> >>>> wxLogError(wxT("%s"), GetLastError().c_str()); >>>> 304 >>>> >>>> } >>>> 305 >>>> >>>> >>>> Oops ! Missed that. Apologies. >>> >>>> >>>> >>>>> >>>>> Will remove the set call and will send you the updated patch if >>>>> everything works fine. >>>>> >>>> >>>> No, we need to ensure the client encoding is set correctly. It just >>>> needs to be set to SQL_ASCII if it's a SQL_ASCII database (I believe). >>>> >>>> >>> Need to rework on the initialise method. Will come with an updated. >>> patch. Sorry for trouble. >>> >>>> >>>>> >>>>> On Tue, Jun 5, 2018 at 6:05 PM, Dave Page wrote: >>>>> >>>>>> Hi >>>>>> >>>>>> On Tue, Jun 5, 2018 at 1:21 PM, Aditya Toshniwal < >>>>>> aditya.toshniwal@enterprisedb.com> wrote: >>>>>> >>>>>>> Hi Dave, >>>>>>> >>>>>>> >>>>>>> On Tue, Jun 5, 2018 at 4:56 PM, Dave Page wrote: >>>>>>> >>>>>>>> Hi >>>>>>>> >>>>>>>> On Tue, Jun 5, 2018 at 9:50 AM, Aditya Toshniwal < >>>>>>>> aditya.toshniwal@enterprisedb.com> wrote: >>>>>>>> >>>>>>>>> Hi Hackers, >>>>>>>>> >>>>>>>>> PFA updated patch. The sqleditor change is sent separately and >>>>>>>>> removed from current patch as suggested. >>>>>>>>> The test cases were running fine when the module was specified >>>>>>>>> using --pkg but were failing in complete run. Fixed that. >>>>>>>>> >>>>>>>> >>>>>>>> I did a quick test by creating a SQL_ASCII database containing a >>>>>>>> simple table: >>>>>>>> >>>>>>>> CREATE TABLE sql_ascii (id serial primary key, data text); >>>>>>>> >>>>>>>> And then populated it with data: >>>>>>>> >>>>>>>> /Library/PostgreSQL/9.4/bin/psql -d sql_ascii -U postgres -c >>>>>>>> "INSERT INTO sql_acsii (data) VALUES ('[Windows-1252] Euro: \x80 Double >>>>>>>> dagger: \x87');" >>>>>>>> /Library/PostgreSQL/9.4/bin/psql -d sql_ascii -U postgres -c >>>>>>>> "INSERT INTO sql_ascii (data) VALUES ('[Latin-1] Yen: \xa5 Half: >>>>>>>> \xbd');" >>>>>>>> /Library/PostgreSQL/9.4/bin/psql -d sql_ascii -U postgres -c >>>>>>>> "INSERT INTO sql_ascii (data) VALUES ('[Japanese] Ship: \xe8\x88\xb9');" >>>>>>>> /Library/PostgreSQL/9.4/bin/psql -d sql_ascii -U postgres -c >>>>>>>> "INSERT INTO sql_ascii (data) VALUES ('[Invalid UTF-8] Blob: >>>>>>>> \xf4\xa5\xa3\xa5');" >>>>>>>> >>>>>>>> I then right-clicked the table in the treeview, and selected the >>>>>>>> option to view all rows, and immediately saw an error: >>>>>>>> >>>>>>>> 2018-06-05 12:23:27,319: SQL pgadmin: Execute (async) for server >>>>>>>> #1 - CONN:1187535 (Query-id: 8522474): >>>>>>>> SELECT * FROM public.sql_ascii >>>>>>>> ORDER BY id ASC >>>>>>>> 2018-06-05 12:23:27,320: ERROR pgadmin: Failed to execute query >>>>>>>> (execute_async) for the server #1 - CONN:1187535(Query-id: 8522474): >>>>>>>> Error Message:ERROR: invalid byte sequence for encoding "UTF8": >>>>>>>> 0x80 >>>>>>>> SQL state: 22021 >>>>>>>> >>>>>>>> Running "SELECT * FROM sql_ascii" in the query tool resulted in the >>>>>>>> same error, however, if I ran "SET client_encoding = 'SQL_ASCII';" first, I >>>>>>>> do see results. >>>>>>>> >>>>>>>> I have confirmed that I've restarted the server after applying the >>>>>>>> patch. >>>>>>>> >>>>>>>> What am I missing? Why don't we just set the client_encoding to >>>>>>>> SQL_ASCII if it's a SQL_ASCII database? >>>>>>>> >>>>>>> >>>>>>> It is by default same as the server encoding. But, the following >>>>>>> existing code in web/pgadmin/utils/driver/psycopg2/connection.py makes >>>>>>> the client_encoding as UNICODE for every connection. I am not sure it >>>>>>> should be removed. >>>>>>> >>>>>>> status = _execute(cur, "SET DateStyle=ISO;" >>>>>>> >>>>>>> "SET client_min_messages=notice;" >>>>>>> >>>>>>> "SET bytea_output=escape;" >>>>>>> >>>>>>> "SET client_encoding='UNICODE';") >>>>>>> >>>>>> >>>>>> It was probably before you joined, but I have said a number of times >>>>>> that pgAdmin 3 handled this differently and that maybe we should do it the >>>>>> same way here. See https://git.postgresql.org >>>>>> /gitweb/?p=pgadmin3.git;a=blob;f=pgadmin/db/pgConn.cpp, in the >>>>>> pgConn::Initialize() function. >>>>>> >>>>>> Either way, your patch isn't working for me. >>>>>> >>>>>> >>>>>>> >>>>>>> >>>>>>> Note that this testing was on Python 2.7.10 on MacOS. >>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> Kindly review. >>>>>>>>> >>>>>>>>> Thanks and Regards, >>>>>>>>> Aditya Toshniwal >>>>>>>>> Software Engineer | EnterpriseDB Software Solutions | Pune >>>>>>>>> "Don't Complain about Heat, Plant a tree" >>>>>>>>> >>>>>>>>> On Tue, Jun 5, 2018 at 10:15 AM, Aditya Toshniwal < >>>>>>>>> aditya.toshniwal@enterprisedb.com> wrote: >>>>>>>>> >>>>>>>>>> Hi >>>>>>>>>> >>>>>>>>>> On Tue, Jun 5, 2018 at 1:08 AM, Joao De Almeida Pereira < >>>>>>>>>> jdealmeidapereira@pivotal.io> wrote: >>>>>>>>>> >>>>>>>>>>> Hello Aditya, >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> There is no change related to notifications in this patch. >>>>>>>>>>>> The below code is minor fix related to connection status of sql >>>>>>>>>>>> editor. Can you please share the code snippet if it is not the below. >>>>>>>>>>>> >>>>>>>>>>>> - # Check for the asynchronous notifies statements. >>>>>>>>>>>> - conn.check_notifies(True) >>>>>>>>>>>> - notifies = conn.get_notifies() >>>>>>>>>>>> + if status is not None: >>>>>>>>>>>> + # Check for the asynchronous notifies statements. >>>>>>>>>>>> + conn.check_notifies(True) >>>>>>>>>>>> + notifies = conn.get_notifies() >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> This is a minor fix, but is it related to querying SQL_ASCII >>>>>>>>>>> database? >>>>>>>>>>> >>>>>>>>>> No its not. It is something I found when I was working on >>>>>>>>>> SQL_ASCII related changes. >>>>>>>>>> Well then, will send a separate patch for it. >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Thanks >>>>>>>>>>> Victoria && Joao >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Dave Page >>>>>>>> Blog: http://pgsnake.blogspot.com >>>>>>>> Twitter: @pgsnake >>>>>>>> >>>>>>>> EnterpriseDB UK: http://www.enterprisedb.com >>>>>>>> The Enterprise PostgreSQL Company >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Dave Page >>>>>> Blog: http://pgsnake.blogspot.com >>>>>> Twitter: @pgsnake >>>>>> >>>>>> EnterpriseDB UK: http://www.enterprisedb.com >>>>>> The Enterprise PostgreSQL Company >>>>>> >>>>> >>>>> >>>> >>>> >>>> -- >>>> Dave Page >>>> Blog: http://pgsnake.blogspot.com >>>> Twitter: @pgsnake >>>> >>>> EnterpriseDB UK: http://www.enterprisedb.com >>>> The Enterprise PostgreSQL Company >>>> >>> >>> >> >> >> -- >> Dave Page >> Blog: http://pgsnake.blogspot.com >> Twitter: @pgsnake >> >> EnterpriseDB UK: http://www.enterprisedb.com >> The Enterprise PostgreSQL Company >> > > -- Dave Page Blog: http://pgsnake.blogspot.com Twitter: @pgsnake EnterpriseDB UK: http://www.enterprisedb.com The Enterprise PostgreSQL Company --000000000000c69f02056e0adeb0 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi

On Wed, Jun 6, 2018 at 2:02 PM, Aditya Toshniwal &l= t;ad= itya.toshniwal@enterprisedb.com> wrote:
Hi Hackers,

<= /div>
PFA updated patch as the previous one was not working as expected= . I have tried to make it similar to that of pgAdmin3 and you do not need t= o change client_encoding as it is set now based on server encoding. It work= s fine with "view data" also.

- In connection.py, at ~409, shouldn't we set the client_encod= ing to SQL_ASCII? Otherwise it could be overridden with something unexpecte= d if the client has PGCLIENTENCODING set for example.

<= div>- With or without that change, I get the following test failure on macO= S with Python 2.7.10:

=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
ERROR: runTest (pgadmin.t= ools.sqleditor.tests.test_encoding_charset.TestEncodingCharset)
W= ith Encoding SQL_ASCII
------------------------------------------= ----------------------------
Traceback (most recent call last):
=C2=A0 File "/Users/dpage/git/pgadmin4/web/pgadmin/tools/sqle= ditor/tests/test_encoding_charset.py", line 86, in runTest
= =C2=A0 =C2=A0 response =3D self.tester.get(url)
=C2=A0 File "= ;/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/werkzeug/te= st.py", line 830, in get
=C2=A0 =C2=A0 return self.open(*arg= s, **kw)
=C2=A0 File "/Users/dpage/.virtualenvs/pgadmin4/lib= /python2.7/site-packages/flask/testing.py", line 127, in open
=C2=A0 =C2=A0 follow_redirects=3Dfollow_redirects)
=C2=A0 File = "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/werkze= ug/test.py", line 803, in open
=C2=A0 =C2=A0 response =3D se= lf.run_wsgi_app(environ, buffered=3Dbuffered)
=C2=A0 File "/= Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/werkzeug/test= .py", line 716, in run_wsgi_app
=C2=A0 =C2=A0 rv =3D run_wsg= i_app(self.application, environ, buffered=3Dbuffered)
=C2=A0 File= "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/werkz= eug/test.py", line 923, in run_wsgi_app
=C2=A0 =C2=A0 app_rv= =3D app(environ, start_response)
=C2=A0 File "/Users/dpage/= .virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py", line = 1997, in __call__
=C2=A0 =C2=A0 return self.wsgi_app(environ, sta= rt_response)
=C2=A0 File "/Users/dpage/.virtualenvs/pgadmin4= /lib/python2.7/site-packages/flask/app.py", line 1985, in wsgi_app
=C2=A0 =C2=A0 response =3D self.handle_exception(e)
=C2=A0= File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/= flask/app.py", line 1540, in handle_exception
=C2=A0 =C2=A0 = reraise(exc_type, exc_value, tb)
=C2=A0 File "/Users/dpage/.= virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py", line 1= 982, in wsgi_app
=C2=A0 =C2=A0 response =3D self.full_dispatch_re= quest()
=C2=A0 File "/Users/dpage/.virtualenvs/pgadmin4/lib/= python2.7/site-packages/flask/app.py", line 1614, in full_dispatch_req= uest
=C2=A0 =C2=A0 rv =3D self.handle_user_exception(e)
=C2=A0 File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-pa= ckages/flask/app.py", line 1517, in handle_user_exception
= =C2=A0 =C2=A0 reraise(exc_type, exc_value, tb)
=C2=A0 File "= /Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py= ", line 1612, in full_dispatch_request
=C2=A0 =C2=A0 rv =3D = self.dispatch_request()
=C2=A0 File "/Users/dpage/.virtualen= vs/pgadmin4/lib/python2.7/site-packages/flask/app.py", line 1598, in d= ispatch_request
=C2=A0 =C2=A0 return self.view_functions[rule.end= point](**req.view_args)
=C2=A0 File "/Users/dpage/.virtualen= vs/pgadmin4/lib/python2.7/site-packages/flask_login.py", line 792, in = decorated_view
=C2=A0 =C2=A0 return func(*args, **kwargs)
=C2=A0 File "/Users/dpage/git/pgadmin4/web/pgadmin/tools/sqleditor/= __init__.py", line 576, in poll
=C2=A0 =C2=A0 'oids'= : oids
=C2=A0 File "/Users/dpage/git/pgadmin4/web/pgadmin/ut= ils/ajax.py", line 61, in make_json_response
=C2=A0 =C2=A0 s= eparators=3D(',', ':')),
=C2=A0 File "/Users= /dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/simplejson/__init_= _.py", line 399, in dumps
=C2=A0 =C2=A0 **kw).encode(obj)
=C2=A0 File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/= site-packages/simplejson/encoder.py", line 291, in encode
= =C2=A0 =C2=A0 chunks =3D self.iterencode(o, _one_shot=3DTrue)
=C2= =A0 File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packag= es/simplejson/encoder.py", line 373, in iterencode
=C2=A0 = =C2=A0 return _iterencode(o, 0)
UnicodeDecodeError: 'utf8'= ; codec can't decode byte 0xad in position 0: invalid start byte
<= div>
--------------------------------------------------------= --------------
Ran 317 tests in 30.692s

= FAILED (errors=3D1, skipped=3D21)

=C2=A0
T= he only problem is, I cannot find equivalent codec for wxConvLibc in python= . The closest one I could find is raw_unicode_escape. So, in a SQL_ASCII da= tabase, non ASCII characters may differ in pgAdmin4 and pgAdmin3, but it wi= ll display results.

Yeah, I thi= nk that's fine. For the small number of people with SQL_ASCII databases= , seeing escaped characters is better than nothing.
=C2=A0
<= blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-l= eft:1px solid rgb(204,204,204);padding-left:1ex">

=

Dave,=C2=A0
You need t= o add "E" before the string to be inserted, otherwise \x will be = considered as a plain string.
IN= SERT INTO sql_ascii (data) VALUES (E'[Invalid UTF-8]=C2=A0 Blob: \xf4\x= a5\xa3\xa5');

Yeah, = sorry - I copied the wrong version of the query :-(
=C2=A0
<= blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-l= eft:1px solid rgb(204,204,204);padding-left:1ex">

=

Kindly review.

Thanks and Regards,
Aditya Toshniwal
Software Engineer |=C2=A0= EnterpriseDB Software Solutions |=C2=A0= Pune
"Don't Compla= in about Heat, Plant a tree"

On Tue, Jun 5, 2018 at 6:42 PM, Dave Page <dpage@pgadm= in.org> wrote:
Hi

On Tue, Jun 5, 2018 at 2:03 PM, Aditya Toshniwal <aditya.toshniwal@enterprisedb.com> wrote:
Hi

On Tue, Jun 5, = 2018 at 6:25 PM, Dave Page <dpage@pgadmin.org> wrote:


On Tue, Jun 5, 201= 8 at 1:49 PM, Aditya Toshniwal <aditya.toshniwal@enterpris= edb.com> wrote:
Hi Dave,

The probl= em of SQL ASCII is solved with the patch, and not related to setting the cl= ient encoding of the sql window.

<= div>No it's not. It doesn't work for me as I said (and showed the e= xample of).

= After setting the client_encoding to SQL_ASCII you got the output. Previous= ly, it used to fail in the back end itself because python encoding failure.= That is fixed.
The error=C2=A0ERR= OR:=C2=A0 invalid byte sequence for encoding "UTF8": 0x80<= /span>=C2=A0is thrown by postgres and not python or pgAdmin4. You will get = the same error even if you=C2=A0
connect from psql.

Sure - but that is not a = fix. You have no way of running the SET command if you're using "v= iew data" - and in the query tool, users just expect it to work (as it= did in pgAdmin 3).
=C2=A0
=C2=A0
I can see there is no SET call in pgAdmin3 for client_encoding= .=C2=A0 I can remove the=C2=A0SET client_encodin= g=3D'UNICODE'; that will solve the= problem. But, can you please let me know why that was added.=C2=A0<= /div>

There is, but it's inside = an API call (PQsetClientEncoding):

<= div>
300 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0wxLogInfo(wxT("Setting=C2=A0client_encoding=C2= =A0to=C2=A0'%s'"),=C2=A0encoding.c_str());
30= 1 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0if=C2=A0(PQsetClientEncoding(conn,=C2=A0encoding.ToAscii()))
302 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0{
303 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0wxLogError(w= xT("%s"),=C2=A0GetLastError().c_str());
304= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0}
305

<= /span>
Oops ! Missed that. Apologies.=C2=A0
=C2=A0

Will remove the set call and will se= nd you the updated patch if everything works fine.

No, we need to ensure the client encoding = is set correctly. It just needs to be set to SQL_ASCII if it's a SQL_AS= CII database (I believe).
=C2=A0
=
Need to rework on the initialise method. Will come with an updated. pa= tch. Sorry for trouble.=C2=A0


On Tue, Jun= 5, 2018 at 6:05 PM, Dave Page <dpage@pgadmin.org> wrote:
Hi
<= div class=3D"gmail_extra">
On Tue, Jun 5, 2018 at 1:21 PM, Aditya Toshniwal = <= aditya.toshniwal@enterprisedb.com> wrote:
Hi Dave,


On Tue, Jun 5, 2018 at 4:56 PM, Dave Page <dpage@pgadmin.org> wrote:
Hi

On Tue, Jun 5, 201= 8 at 9:50 AM, Aditya Toshniwal <aditya.toshniwal@enterpris= edb.com> wrote:
Hi Hackers,

PFA updated pa= tch. The sqleditor change is sent separately and removed from current patch= as suggested.
The test cases were running fine when the module w= as specified using --pkg but were failing in complete run. Fixed that.

I did a quick test by creati= ng a SQL_ASCII database containing a simple table:

CREATE TABLE sql_ascii (id serial primary key, data text);

<= /div>
And then populated it with data:

/L= ibrary/PostgreSQL/9.4/bin/psql -d sql_ascii -U postgres -c "INSER= T INTO sql_acsii (data) VALUES ('[Windows-1252]=C2=A0 =C2=A0Euro: \x80= =C2=A0 =C2=A0Double dagger: \x87');"
/Library/PostgreSQL= /9.4/bin/psql -d sql_ascii -U postgres -c "INSERT INTO sql_ascii = (data) VALUES ('[Latin-1]=C2=A0 =C2=A0Yen: \xa5=C2=A0 =C2=A0Half: \xbd&= #39;);"
/Library/PostgreSQL/9.4/bin/psql -d sql_ascii -= U postgres -c "INSERT INTO sql_ascii (data) VALUES ('[Japanese]=C2= =A0 =C2=A0Ship: \xe8\x88\xb9');"
/Library/PostgreSQL/9.4= /bin/psql -d sql_ascii -U postgres -c "INSERT INTO sql_ascii (dat= a) VALUES ('[Invalid UTF-8]=C2=A0 Blob: \xf4\xa5\xa3\xa5');"

I then right-clicked the table in the treevie= w, and selected the option to view all rows, and immediately saw an error:<= /div>

2018-06-05 12:23:27,319: SQL pgadmin: <= /span>Execute (async) for server #1 - CONN:1187535 (Query-id: 8522474):
SELECT * FROM public.sql_ascii
ORDER BY id ASC=C2=A0
=
2018-06-05 12:23:27,320: ERROR pgadmin: Failed to execute= query (execute_async) for the server #1 - CONN:1187535(Query-id: 8522474):=
Error Message:ERROR:=C2=A0 invalid byte sequence for encoding &q= uot;UTF8": 0x80
SQL state: 22021

<= div>Running "SELECT * FROM sql_ascii" in the query tool resulted = in the same error, however, if I ran "SET client_encoding =3D 'SQL= _ASCII';" first, I do see results.

I have= confirmed that I've restarted the server after applying the patch.

What am I missing? Why don't we just set the clie= nt_encoding to SQL_ASCII if it's a SQL_ASCII database?
=C2=A0
It is by default same as t= he server encoding. But, the following existing code in=C2=A0=C2=A0web/pgadmin/utils/driver/psycopg2/connection.py makes = the client_encoding as UNICODE for every connection. I am not sure it shoul= d be removed.

<= span class=3D"gmail-m_-5498328813472628049gmail-m_584540487799665668gmail-m= _-7949188697271137504m_-2889938182068220083m_-8242036136222913340gmail-m_33= 97243500728641691m_6815941991577971575gmail-m_-5207798786893497198gmail-s1"= style=3D"font-variant-ligatures:no-common-ligatures">= =C2=A0 =C2=A0 =C2=A0 =C2=A0 status =3D _execute(cur, "SET DateS= tyle=3DISO;"

<= span class=3D"gmail-m_-5498328813472628049gmail-m_584540487799665668gmail-m= _-7949188697271137504m_-2889938182068220083m_-8242036136222913340gmail-m_33= 97243500728641691m_6815941991577971575gmail-m_-5207798786893497198gmail-s1"= style=3D"font-variant-ligatures:no-common-ligatures">= =C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 "SET client_min_messages=3D= notice;"

<= span class=3D"gmail-m_-5498328813472628049gmail-m_584540487799665668gmail-m= _-7949188697271137504m_-2889938182068220083m_-8242036136222913340gmail-m_33= 97243500728641691m_6815941991577971575gmail-m_-5207798786893497198gmail-s1"= style=3D"font-variant-ligatures:no-common-ligatures">= =C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 "SET bytea_output=3Descape;= "

<= span class=3D"gmail-m_-5498328813472628049gmail-m_584540487799665668gmail-m= _-7949188697271137504m_-2889938182068220083m_-8242036136222913340gmail-m_33= 97243500728641691m_6815941991577971575gmail-m_-5207798786893497198gmail-s1"= style=3D"font-variant-ligatures:no-common-ligatures">= =C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 "SET client_encoding=3D'= ;UNICODE';")


It was probably before you joined, but I h= ave said a number of times that pgAdmin 3 handled this differently and that= maybe we should do it the same way here. See=C2=A0https://git.postgresql.org/gitweb/?p=3Dpgadmin3.git;= a=3Dblob;f=3Dpgadmin/db/pgConn.cpp, in the pgConn::Initialize() fu= nction.

Either way, your patch isn't working f= or me.
=C2=A0
<= div class=3D"gmail_quote">

=
Note that this testing was on Python 2.7.10 on MacOS.
=C2=A0

Kindly review.

= Thanks= and Regards,
= Aditya= Toshniwal
Software Engineer |=C2=A0EnterpriseDB Software Solutions |=C2=A0= Pune
"Don't Complain about Heat, Plant a tree"

On Tue, J= un 5, 2018 at 10:15 AM, Aditya Toshniwal <aditya.toshniwal= @enterprisedb.com> wrote:
Hi

On Tue, Jun 5, 2018 at 1:08 AM, Joao De Alme= ida Pereira <jdealmeidapereira@pivotal.io> wrote:=
Hell= o Aditya,


There is no c= hange related to notifications in this patch.=C2=A0
The below cod= e is minor fix related to connection status of sql editor. Can you please s= hare the code snippet if it is not the below.

-=C2=A0 =C2=A0 =C2=A0 =C2=A0 # Check for the asynchronous notifies stateme= nts.
-=C2=A0 =C2=A0 =C2=A0 =C2=A0 conn.check_notifies(True)
=
-=C2=A0 =C2=A0 =C2=A0 =C2=A0 notifies =3D conn.get_notifies()
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 if status is not None:
+=C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 # Check for the asynchronous notifies state= ments.
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 conn.check_noti= fies(True)
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 notifies = =3D conn.get_notifies()

<= br>
This is a minor fix, but is it related to querying SQL= _ASCII database?
No its not. It i= s something I found when I was working on SQL_ASCII related changes.
<= div>Well then, will send a separate patch for it.
<= div>
Thanks
Victoria && Joao





--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter= : @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise Postgre= SQL Company




--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

Enterpr= iseDB UK: http://= www.enterprisedb.com
The Enterprise PostgreSQL Company





--
Dave Page
Blog: http://pgsnake.blogspot.com=
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Ent= erprise PostgreSQL Company




--
Dave Page
Blog: http://pgsnake.blogspot.= com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The = Enterprise PostgreSQL Company
--000000000000c69f02056e0adeb0--