Subject: [pgAdmin4][RM#3289] Can't query SQL_ASCII database.

From: Aditya Toshniwal <[email-masked]>
Date: Tue, 22 May 2018 11:35:32 +0530

Hi Hackers,

PFA patch for RM#3289 where decode error was thrown on querying a SQL_ASCII
database table. Please note, this problem occurs only on windows.
Sample insert - insert into test_tab values ('é');

psycopg2 has a encodings dictionary where Postgres Database Encodings are
mapped to python equivalent. It uses 'ascii' decoder of python to decode
for SQL_ASCII encoding. If data has characters beyond the limit of ascii
then it failed. The solution would be to use utf_8 decoder instead of
ascii. I tried setting the client_encoding using
set_client_encoding('UTF8') method of a psycopg2 connection but no luck
(also its not allowed for async connection). I also tried executing "SET
CLIENT_ENCODING='UTF8'" but it didn't work too.
So, as in the patch, I had to set encodings dict value directly to 'utf_8'
and it seems to be working. Please note, the same is added to psycopg3
milestones
https://github.com/psycopg/psycopg2/milestone/4

Also fixed a small glitch for sql editor connection status check.

Kindly review.

Thanks and Regards,
Aditya Toshniwal
Software Engineer | EnterpriseDB Software Solutions | Pune
"Don't Complain about Heat, Plant a tree"

---

From: Dave Page <[email-masked]>
Date: Tue, 22 May 2018 09:07:56 +0100

Hi

Pivotal's buildbot is showing problems with this patch:

https://gpdb-dev.bosh.pivotalci.info/teams/pgadmin/pipelines/pgadmin-patch/jobs/run-linter/builds/66
(linter failed)
https://gpdb-dev.bosh.pivotalci.info/teams/pgadmin/pipelines/pgadmin-patch/jobs/run-tests/builds/84
(tests failed)

On Tue, May 22, 2018 at 7:05 AM, Aditya Toshniwal <
[email-masked]> wrote:

> Hi Hackers,
>
> PFA patch for RM#3289 where decode error was thrown on querying a
> SQL_ASCII database table. Please note, this problem occurs only on windows.

-- 
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

---

From: Aditya Toshniwal <[email-masked]>
Date: Tue, 22 May 2018 14:16:21 +0530

Hi,

PFA updated patch. Linter issues are fixed ( we dont have any linter setup
for python :-( )
Regarding test cases, they run successfully on my system and the reason it
failed for pivotal is timeout exception. I am sorry I can't help with that.

Traceback (most recent call last):
  File
"/tmp/build/a453582b/pgadmin-repo/web/pgadmin/feature_tests/keyboard_shortcut_test.py",
line 52, in runTest
    self._check_shortcuts()
  File
"/tmp/build/a453582b/pgadmin-repo/web/pgadmin/feature_tests/keyboard_shortcut_test.py",
line 77, in _check_shortcuts
    ") and contains(@class, 'open')]")
  File
"/root/.pyenv/versions/pgadmin36/lib/python3.6/site-packages/selenium/webdriver/support/wait.py",
line 80, in until
    raise TimeoutException(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message:

Thanks and Regards,
Aditya Toshniwal

On Tue, May 22, 2018 at 1:37 PM, Dave Page <[email-masked]> wrote:

> https://gpdb-dev.bosh.pivotalci.info/teams/pgadmin/
> pipelines/pgadmin-patch/jobs/run-linter/builds/66 (linter failed)
> https://gpdb-dev.bosh.pivotalci.info/teams/pgadmin/
> pipelines/pgadmin-patch/jobs/run-tests/builds/84 (tests failed)
>
>
> On Tue, May 22, 2018 at 7:05 AM, Aditya Toshniwal <aditya.toshniwal@
> enterprisedb.com> wrote:
>
>> Hi Hackers,
>>
>> So, as in the patch, I had to set encodings dict value directly to
>> 'utf_8' and it seems to be working. Please note, the same is added to
>> psycopg3 milestones
diff --git a/web/pgadmin/tools/sqleditor/__init__.py b/web/pgadmin/tools/sqleditor/__init__.py
index c72505a..fa8b83f 100644
--- a/web/pgadmin/tools/sqleditor/__init__.py
+++ b/web/pgadmin/tools/sqleditor/__init__.py
@@ -1474,7 +1474,8 @@ def query_tool_status(trans_id):
             errormsg=error_msg
         )
 
-    if conn and trans_obj and session_obj:
+    if conn and trans_obj and session_obj \
+        and conn.transaction_status() is not None:
         status = conn.transaction_status()
         return make_json_response(
             data={
diff --git a/web/pgadmin/utils/driver/psycopg2/connection.py b/web/pgadmin/utils/driver/psycopg2/connection.py
index 315631c..8ca30b3 100644
--- a/web/pgadmin/utils/driver/psycopg2/connection.py
+++ b/web/pgadmin/utils/driver/psycopg2/connection.py
@@ -49,6 +49,10 @@ else:
 
 _ = gettext
 
+# Use utf_8 decoder instead of psycopg2 default ascii for
+# SQL_ASCII database to avoid decode errors
+encodings['SQL_ASCII'] = 'utf_8'
+encodings['SQLASCII'] = 'utf_8'
 
 # Register global type caster which will be applicable to all connections.
 register_global_typecasters()

---

From: Victoria Henry <[email-masked]>
Date: Tue, 22 May 2018 09:45:44 -0400

Hi Aditya,

We made a minor change to make the patch so the python linter can pass.
Attached is the change we made.
Everything else looks good.

Sincerely,

Victoria & Anthony

On Tue, May 22, 2018 at 4:46 AM Aditya Toshniwal <
[email-masked]> wrote:

>>> psycopg2 has a encodings dictionary where Postgres Database Encodings
>>> are mapped to python equivalent. It uses 'ascii' decoder of python to
>>> decode for SQL_ASCII encoding. If data has characters beyond the limit of
>>> ascii then it failed. The solution would be to use utf_8 decoder instead of

---

From: Aditya Toshniwal <[email-masked]>
Date: Wed, 23 May 2018 10:13:27 +0530

Thank you Victoria, Anthony.

Thanks and Regards,
Aditya Toshniwal

On Tue, May 22, 2018 at 7:15 PM, Victoria Henry <[email-masked]> wrote:

> On Tue, May 22, 2018 at 4:46 AM Aditya Toshniwal <aditya.toshniwal@
> enterprisedb.com> wrote:
>
>> Hi,
>>
>> PFA updated patch. Linter issues are fixed ( we dont have any linter
>> setup for python :-( )
>> Regarding test cases, they run successfully on my system and the reason
>> it failed for pivotal is timeout exception. I am sorry I can't help with
>> that.
>>
>>   File "/tmp/build/a453582b/pgadmin-repo/web/pgadmin/feature_
>> tests/keyboard_shortcut_test.py", line 52, in runTest
>>     self._check_shortcuts()
>>   File "/tmp/build/a453582b/pgadmin-repo/web/pgadmin/feature_
>> tests/keyboard_shortcut_test.py", line 77, in _check_shortcuts
>>   File "/root/.pyenv/versions/pgadmin36/lib/python3.6/site-
>> packages/selenium/webdriver/support/wait.py", line 80, in until

---

From: Aditya Toshniwal <[email-masked]>
Date: Wed, 30 May 2018 12:36:32 +0530

Hi Hackers,

PFA updated patch after all the permutations, combinations for encoding for
SQL_ASCII database.  Also fixed a small glitch for sql editor connection
status check.

Please note, ERROR: invalid byte sequence for encoding "UTF8": 0xe9 0x73 is
a Postgres DB error and not pgAdmin4 error.

You need to change client_encoding to the appropriate. After changing
client_encoding using command - set client_encoding='XYZ', it will give not
give error.

Thanks and Regards,
Aditya Toshniwal

On Wed, May 23, 2018 at 10:13 AM, Aditya Toshniwal <
[email-masked]> wrote:

>>>   File "/tmp/build/a453582b/pgadmin-repo/web/pgadmin/feature_tests/keyboard_shortcut_test.py",
>>> line 52, in runTest
>>>     self._check_shortcuts()
>>>   File "/tmp/build/a453582b/pgadmin-repo/web/pgadmin/feature_tests/keyboard_shortcut_test.py",
>>>   File "/root/.pyenv/versions/pgadmin36/lib/python3.6/site-packages
>>> /selenium/webdriver/support/wait.py", line 80, in until
>>>> https://gpdb-dev.bosh.pivotalci.info/teams/pgadmin/pipelines
>>>> /pgadmin-patch/jobs/run-linter/builds/66 (linter failed)
>>>> https://gpdb-dev.bosh.pivotalci.info/teams/pgadmin/pipelines
>>>> /pgadmin-patch/jobs/run-tests/builds/84 (tests failed)
>>>>
>>>>

---

From: Joao De Almeida Pereira <[email-masked]>
Date: Wed, 30 May 2018 16:36:24 -0400

Hello Aditya,

It looks ok and it passes CI.

We have some recommendations:
- These look like 2 different changes so they should be in separated commits
- Do we have test coverage for the bug that you are talking about? If not
we should, to ensure this problem does not happen again in a future change.

Thanks
Victoria && Joao

On Wed, May 30, 2018 at 3:06 AM Aditya Toshniwal <
[email-masked]> wrote:

> Hi Hackers,
>
> PFA updated patch after all the permutations, combinations for encoding
> for SQL_ASCII database.  Also fixed a small glitch for sql editor
> connection status check.
>
> Please note, ERROR: invalid byte sequence for encoding "UTF8": 0xe9 0x73
> is a Postgres DB error and not pgAdmin4 error.
>
>
>

---

From: Aditya Toshniwal <[email-masked]>
Date: Thu, 31 May 2018 10:50:19 +0530

Hi Victoria/Joao,

> - These look like 2 different changes so they should be in separated

If you are talking of set client_encoding, then its not a bug. Its a choice
given to Postgres DB user to change the encoding of the characters.
Postgres will translate characters from Server Encoding to Client Encoding,
and will throw error like mentioned previously. This link will help better
- https://www.postgresql.org/docs/10/static/multibyte.html
The actual bug was, even after changing the client encoding to SQL_ASCII,
pgAdmin4 was not able to show the output as it was failing in encoding by
psycopg2. The patch is for resolving that.

> we should, to ensure this problem does not happen again in a future change.

It is not possible adding test cases for encoding related stuff, as
Postgres support a lot many different types of encoding and conversions
(refer above link)

>
> Thanks
> Victoria && Joao
>
> On Wed, May 30, 2018 at 3:06 AM Aditya Toshniwal <aditya.toshniwal@
> enterprisedb.com> wrote:
>
>> Hi Hackers,
>>
>> <Image Deleted>
>> <Image Deleted>
>> On Wed, May 23, 2018 at 10:13 AM, Aditya Toshniwal <aditya.toshniwal@
>> enterprisedb.com> wrote:
>>
>>>> We made a minor change to make the patch so the python linter can
>>>> pass.  Attached is the change we made.
>>>>> Regarding test cases, they run successfully on my system and the
>>>>> reason it failed for pivotal is timeout exception. I am sorry I can't help
>>>>> with that.
>>>>>
>>>>>>> psycopg2 has a encodings dictionary where Postgres Database
>>>>>>> Encodings are mapped to python equivalent. It uses 'ascii' decoder of
>>>>>>> python to decode for SQL_ASCII encoding. If data has characters beyond the
>>>>>>> limit of ascii then it failed. The solution would be to use utf_8 decoder
>>>>>>> instead of ascii. I tried setting the client_encoding using

---

From: Dave Page <[email-masked]>
Date: Thu, 31 May 2018 09:12:59 -0400

Hi

On Thu, May 31, 2018 at 1:20 AM, Aditya Toshniwal <
[email-masked]> wrote:

> If you are talking of set client_encoding, then its not a bug. Its a
> choice given to Postgres DB user to change the encoding of the characters.

I was going to ask the same thing. Per
https://www.postgresql.org/docs/10/static/multibyte.html#id-1.6.10.5.7,
every characterset except SQL_ASCII can be converted to UTF-8, so we only
need to test that UTF-8 and some other charactersets besides SQL_ASCII
work, and then separately that SQL_ASCII with characters known not to be in
UTF-8 work.

---

From: Aditya Toshniwal <[email-masked]>
Date: Mon, 4 Jun 2018 10:53:20 +0530

Hi Hackers,

PFA the updated patch on this. I have tried to add test cases to check for
different encoding database. In the test run, it will create a database,
fire a query for a string, check if we get the output and drops the
database.
Kindly review.

Thanks and Regards,
Aditya Toshniwal

On Thu, May 31, 2018 at 6:42 PM, Dave Page <[email-masked]> wrote:

> Hi
>
> On Thu, May 31, 2018 at 1:20 AM, Aditya Toshniwal <aditya.toshniwal@
> enterprisedb.com> wrote:
>
>> Hi Victoria/Joao,
>>
>>> - Do we have test coverage for the bug that you are talking about? If
>>> not we should, to ensure this problem does not happen again in a future
>>> change.
>>>
>>
> I was going to ask the same thing. Per https://www.postgresql.
> org/docs/10/static/multibyte.html#id-1.6.10.5.7, every characterset
> except SQL_ASCII can be converted to UTF-8, so we only need to test that
> UTF-8 and some other charactersets besides SQL_ASCII work, and then
> separately that SQL_ASCII with characters known not to be in UTF-8 work.
>
>
>>
>>> Thanks
>>> Victoria && Joao
>>>
>>>> Please note, ERROR: invalid byte sequence for encoding "UTF8": 0xe9
>>>> 0x73 is a Postgres DB error and not pgAdmin4 error.
>>>>
>>>> <Image Deleted>
>>>>
diff --git a/web/pgadmin/tools/sqleditor/__init__.py b/web/pgadmin/tools/sqleditor/__init__.py
index 013e4dc..a9460dd 100644
--- a/web/pgadmin/tools/sqleditor/__init__.py
+++ b/web/pgadmin/tools/sqleditor/__init__.py
@@ -1480,19 +1480,24 @@ def query_tool_status(trans_id):
     if conn and trans_obj and session_obj:
         status = conn.transaction_status()
 
-        # Check for the asynchronous notifies statements.
-        conn.check_notifies(True)
-        notifies = conn.get_notifies()
+        if status is not None:
+            # Check for the asynchronous notifies statements.
+            conn.check_notifies(True)
+            notifies = conn.get_notifies()
 
-        return make_json_response(
-            data={
-                'status': status,
-                'message': gettext(
-                    CONNECTION_STATUS_MESSAGE_MAPPING.get(status),
-                ),
-                'notifies': notifies
-            }
-        )
+            return make_json_response(
+                data={
+                    'status': status,
+                    'message': gettext(
+                        CONNECTION_STATUS_MESSAGE_MAPPING.get(status),
+                    ),
+                    'notifies': notifies
+                }
+            )
+        else:
+            return internal_server_error(
+                errormsg=gettext("Transaction status check failed.")
+            )
     else:
         return internal_server_error(
             errormsg=gettext("Transaction status check failed.")
diff --git a/web/pgadmin/utils/driver/psycopg2/connection.py b/web/pgadmin/utils/driver/psycopg2/connection.py
index cfd161a..e8ca886 100644
--- a/web/pgadmin/utils/driver/psycopg2/connection.py
+++ b/web/pgadmin/utils/driver/psycopg2/connection.py
@@ -51,6 +51,12 @@ else:
 _ = gettext

+# Replace default ascii encoder with unicode-escape
+# which translates characters to unicode format.
+# Escape special characters to ASCII based on unicode
+encodings['SQL_ASCII'] = 'unicode-escape'
+encodings['SQLASCII'] = 'unicode-escape'
+
 # Register global type caster which will be applicable to all connections.
 register_global_typecasters()
 
diff --git a/web/pgadmin/utils/driver/psycopg2/typecast.py b/web/pgadmin/utils/driver/psycopg2/typecast.py
index f136604..ccebb2f 100644
--- a/web/pgadmin/utils/driver/psycopg2/typecast.py
+++ b/web/pgadmin/utils/driver/psycopg2/typecast.py
@@ -164,7 +164,7 @@ def register_global_typecasters():

 def register_string_typecasters(connection):
-    if connection.encoding != 'UTF8':
+    if connection.encoding not in ('UTF8', 'SQLASCII', 'SQL_ASCII'):
         # In python3 when database encoding is other than utf-8 and client
         # encoding is set to UNICODE then we need to map data from database
         # encoding to utf-8.
diff --git a/web/regression/python_test_utils/test_utils.py b/web/regression/python_test_utils/test_utils.py
index 3e517b6..163af10 100644
--- a/web/regression/python_test_utils/test_utils.py
+++ b/web/regression/python_test_utils/test_utils.py
@@ -116,7 +116,7 @@ def clear_node_info_dict():
         del node_info_dict[node][:]

-def create_database(server, db_name):
+def create_database(server, db_name, encoding=None):
     """This function used to create database and returns the database id"""
     try:
         connection = get_db_connection(
@@ -130,8 +130,14 @@ def create_database(server, db_name):
         old_isolation_level = connection.isolation_level
         connection.set_isolation_level(0)
         pg_cursor = connection.cursor()
-        pg_cursor.execute(
-            '''CREATE DATABASE "%s" TEMPLATE template0''' % db_name)
+        if encoding is None:
+            pg_cursor.execute(
+                '''CREATE DATABASE "%s" TEMPLATE template0''' % db_name)
+        else:
+            pg_cursor.execute(
+                '''CREATE DATABASE "%s" TEMPLATE template0 
+                ENCODING='%s' LC_COLLATE='%s' LC_CTYPE='%s' ''' %
+                (db_name, encoding[0], encoding[1], encoding[1]))
         connection.set_isolation_level(old_isolation_level)
         connection.commit()

---

From: Aditya Toshniwal <[email-masked]>
Date: Mon, 4 Jun 2018 15:24:56 +0530

Hi Hackers,

Please ignore previous patch. Fixed some linter issues. PFA updated patch.

Thanks and Regards,
Aditya Toshniwal

On Mon, Jun 4, 2018 at 10:53 AM, Aditya Toshniwal <
[email-masked]> wrote:

>>> The actual bug was, even after changing the client encoding to
>>> SQL_ASCII, pgAdmin4 was not able to show the output as it was failing in
>>> encoding by psycopg2. The patch is for resolving that.
>>>
>>>
>> I was going to ask the same thing. Per https://www.postgresql.org
>> /docs/10/static/multibyte.html#id-1.6.10.5.7, every characterset except
>> SQL_ASCII can be converted to UTF-8, so we only need to test that UTF-8 and
>> some other charactersets besides SQL_ASCII work, and then separately that
>> SQL_ASCII with characters known not to be in UTF-8 work.
>>
>>
>>>
>>>> Thanks
>>>> Victoria && Joao
>>>>
>>>>> PFA updated patch after all the permutations, combinations for
>>>>> encoding for SQL_ASCII database.  Also fixed a small glitch for sql
>>>>> editor connection status check.
>>>>>
>>>>>>>> PFA updated patch. Linter issues are fixed ( we dont have any
>>>>>>>> linter setup for python :-( )
>>>>>>>>>> So, as in the patch, I had to set encodings dict value directly
>>>>>>>>>> to 'utf_8' and it seems to be working. Please note, the same is added to
>>>>>>>>>> psycopg3 milestones

-        return make_json_response(
-            data={
-                'status': status,
-                'message': gettext(
-                    CONNECTION_STATUS_MESSAGE_MAPPING.get(status),
-                ),
-                'notifies': notifies
-            }
-        )
+            return make_json_response(
+                data={
+                    'status': status,
+                    'message': gettext(
+                        CONNECTION_STATUS_MESSAGE_MAPPING.get(status),
+                    ),
+                    'notifies': notifies
+                }
+            )
+        else:
+            return internal_server_error(
+                errormsg=gettext("Transaction status check failed.")
+            )
     else:
         return internal_server_error(
             errormsg=gettext("Transaction status check failed.")
diff --git a/web/pgadmin/tools/sqleditor/tests/test_encoding_charset.py b/web/pgadmin/tools/sqleditor/tests/test_encoding_charset.py
new file mode 100644
index 0000000..64155bf
--- /dev/null
+++ b/web/pgadmin/tools/sqleditor/tests/test_encoding_charset.py
@@ -0,0 +1,132 @@
+##########################################################################
+#
+# pgAdmin 4 - PostgreSQL Tools
+#
+# Copyright (C) 2013 - 2018, The pgAdmin Development Team
+# This software is released under the PostgreSQL Licence
+#
+##########################################################################
+
+import os
+import re
+
+from flask import Flask, render_template
+from jinja2 import FileSystemLoader
+
+from pgadmin import VersionedTemplateLoader
+from pgadmin.utils.route import BaseTestGenerator
+from pgadmin.utils.driver import get_driver
+from config import PG_DEFAULT_DRIVER
+from pgadmin.browser.server_groups.servers.databases.tests import utils as \
+    database_utils
+
+from regression import parent_node_dict
+from regression.python_test_utils import test_utils
+import json
+
+
+class TestEncodingCharset(BaseTestGenerator):
+    """
+    This class validates character support in pgAdmin4 for
+    different PostgresDB encodings
+    """
+    scenarios = [
+        (
+            'With Encoding UTF8',
+            dict(
+                db_encoding='UTF8',
+                lc_collate='C',
+                test_str='A',
+                set_client_encoding=False
+            )),
+        (
+            'With Encoding WIN1252',
+            dict(
+                db_encoding='WIN1252',
+                lc_collate='C',
+                test_str='A',
+                set_client_encoding=False
+            )),
+        (
+            'With Encoding EUC_CN',
+            dict(
+                db_encoding='EUC_CN',
+                lc_collate='C',
+                test_str='A',
+                set_client_encoding=False
+            )),
+        (
+            'With Encoding SQL_ASCII',
+            dict(
+                db_encoding='SQL_ASCII',
+                lc_collate='C',
+                test_str='\\255',
+                set_client_encoding=True
+            )),
+    ]
+
+    def setUp(self):
+        server_details = parent_node_dict['server'][-1]
+        server = server_details['server']
+        self.main_conn = test_utils.get_db_connection(
+            server['db'],
+            server['username'],
+            server['db_password'],
+            server['host'],
+            server['port'],
+            server['sslmode']
+        )
+        self.encode_db_name = 'encoding_test'
+        self.encode_sid = server_details['server_id']
+        self.encode_did = test_utils.create_database(
+            self.server, self.encode_db_name,
+            (self.db_encoding, self.lc_collate))
+
+    def runTest(self):
+
+        db_con = database_utils.connect_database(self,
+                                                 test_utils.SERVER_GROUP,
+                                                 self.encode_sid,
+                                                 self.encode_did)
+        if not db_con["info"] == "Database connected.":
+            raise Exception("Could not connect to the database.")
+
+        # Initialize query tool
+        url = '/datagrid/initialize/query_tool/{0}/{1}/{2}'.format(
+            test_utils.SERVER_GROUP, self.encode_sid, self.encode_did)
+        response = self.tester.post(url)
+        self.assertEquals(response.status_code, 200)
+
+        response_data = json.loads(response.data.decode('utf-8'))
+        self.trans_id = response_data['data']['gridTransId']
+
+        if self.set_client_encoding:
+            # Change Client Encoding
+            url = "/sqleditor/query_tool/start/{0}".format(self.trans_id)
+            sql = "set client_encoding = '{0}';".format(self.db_encoding)
+            response = self.tester.post(url, data=json.dumps({"sql": sql}),
+                                        content_type='html/json')
+            self.assertEquals(response.status_code, 200)
+            url = '/sqleditor/poll/{0}'.format(self.trans_id)
+            response = self.tester.get(url)
+            self.assertEquals(response.status_code, 200)
+            response_data = json.loads(response.data.decode('utf-8'))
+
+        # Check character
+        url = "/sqleditor/query_tool/start/{0}".format(self.trans_id)
+        sql = "select E'{0}';".format(self.test_str)
+        response = self.tester.post(url, data=json.dumps({"sql": sql}),
+                                    content_type='html/json')
+        self.assertEquals(response.status_code, 200)
+        url = '/sqleditor/poll/{0}'.format(self.trans_id)
+        response = self.tester.get(url)
+        self.assertEquals(response.status_code, 200)
+        response_data = json.loads(response.data.decode('utf-8'))
+        self.assertEquals(response_data['data']['rows_fetched_to'], 1)
+
+        database_utils.disconnect_database(self, self.encode_sid,
+                                           self.encode_did)
+
+    def tearDown(self):
+        test_utils.drop_database(self.main_conn, self.encode_db_name)
+        self.main_conn.close()
diff --git a/web/pgadmin/utils/driver/psycopg2/connection.py b/web/pgadmin/utils/driver/psycopg2/connection.py
index cfd161a..e8ca886 100644
--- a/web/pgadmin/utils/driver/psycopg2/connection.py
+++ b/web/pgadmin/utils/driver/psycopg2/connection.py
@@ -51,6 +51,12 @@ else:
 _ = gettext

index 3e517b6..464a09e 100644

---

From: Joao De Almeida Pereira <[email-masked]>
Date: Mon, 4 Jun 2018 11:24:18 -0400

Hi Aditya,
Looks like there are changes in this patch that are related to
notifications, these should go into a separate commit as they are not
related to encoding.
Also the tests are failing in our pipeline:
https://gpdb-dev.bosh.pivotalci.info/teams/pgadmin/pipelines/pgadmin-patch/jobs/run-tests/builds/110

Thanks
Victoria && Joao

On Mon, Jun 4, 2018 at 5:55 AM Aditya Toshniwal <
[email-masked]> wrote:

>> PFA the updated patch on this. I have tried to add test cases to check
>> for different encoding database. In the test run, it will create a
>> database, fire a query for a string, check if we get the output and drops
>> the database.
>> Kindly review.
>>
>> Thanks and Regards,
>> Aditya Toshniwal
>>>>>>>>>>> PFA patch for RM#3289 where decode error was thrown on querying
>>>>>>>>>>> a SQL_ASCII database table. Please note, this problem occurs only on
>>>>>>>>>>> windows.

---

From: Aditya Toshniwal <[email-masked]>
Date: Mon, 4 Jun 2018 22:14:19 +0530

Hi Victoria/Joao,

There is no change related to notifications in this patch.
The below code is minor fix related to connection status of sql editor. Can
you please share the code snippet if it is not the below.

The test cases work perfectly fine on my machine. I will try if I can
simulate the failure on some other machines.

Thanks and Regards,
Aditya Toshniwal

On Mon, Jun 4, 2018 at 8:54 PM, Joao De Almeida Pereira <
[email-masked]> wrote:

> Also the tests are failing in our pipeline: https://gpdb-dev.
> bosh.pivotalci.info/teams/pgadmin/pipelines/pgadmin-
> patch/jobs/run-tests/builds/110
>
> Thanks
> Victoria && Joao
>
> On Mon, Jun 4, 2018 at 5:55 AM Aditya Toshniwal <aditya.toshniwal@
> enterprisedb.com> wrote:
>
>> Hi Hackers,
>>
>> On Mon, Jun 4, 2018 at 10:53 AM, Aditya Toshniwal <aditya.toshniwal@
>> enterprisedb.com> wrote:
>>
>>> Hi Hackers,
>>>
>>>>>>> You need to change client_encoding to the appropriate. After
>>>>>>> changing client_encoding using command - set client_encoding='XYZ', it will
>>>>>>> give not give error.
>>>>>>>
>>>>>>> <Image Deleted>
>>>>>>>
>>>>>>> Thanks and Regards,
>>>>>>> Aditya Toshniwal

---

From: Joao De Almeida Pereira <[email-masked]>
Date: Mon, 4 Jun 2018 15:38:25 -0400

Hello Aditya,

> The below code is minor fix related to connection status of sql editor.
> Can you please share the code snippet if it is not the below.
>
This is a minor fix, but is it related to querying SQL_ASCII database?

Thanks
Victoria && Joao

---

From: Aditya Toshniwal <[email-masked]>
Date: Tue, 5 Jun 2018 10:15:33 +0530

Hi

> This is a minor fix, but is it related to querying SQL_ASCII database?
No its not. It is something I found when I was working on SQL_ASCII related
changes.
Well then, will send a separate patch for it.

---

From: Aditya Toshniwal <[email-masked]>
Date: Tue, 5 Jun 2018 14:20:20 +0530

Hi Hackers,

PFA updated patch. The sqleditor change is sent separately and removed from
current patch as suggested.
The test cases were running fine when the module was specified using --pkg
but were failing in complete run. Fixed that.

Kindly review.

Thanks and Regards,
Aditya Toshniwal

On Tue, Jun 5, 2018 at 10:15 AM, Aditya Toshniwal <
[email-masked]> wrote:

> No its not. It is something I found when I was working on SQL_ASCII
> related changes.

---

From: Dave Page <[email-masked]>
Date: Tue, 5 Jun 2018 12:26:23 +0100

Hi

On Tue, Jun 5, 2018 at 9:50 AM, Aditya Toshniwal <
[email-masked]> wrote:

> Hi Hackers,
>
> PFA updated patch. The sqleditor change is sent separately and removed
> from current patch as suggested.

I did a quick test by creating a SQL_ASCII database containing a simple
table:

CREATE TABLE sql_ascii (id serial primary key, data text);

And then populated it with data:

/Library/PostgreSQL/9.4/bin/psql -d sql_ascii -U postgres -c "INSERT INTO
sql_acsii (data) VALUES ('[Windows-1252]   Euro: \x80   Double dagger:
\x87');"
/Library/PostgreSQL/9.4/bin/psql -d sql_ascii -U postgres -c "INSERT INTO
sql_ascii (data) VALUES ('[Latin-1]   Yen: \xa5   Half: \xbd');"
/Library/PostgreSQL/9.4/bin/psql -d sql_ascii -U postgres -c "INSERT INTO
sql_ascii (data) VALUES ('[Japanese]   Ship: \xe8\x88\xb9');"
/Library/PostgreSQL/9.4/bin/psql -d sql_ascii -U postgres -c "INSERT INTO
sql_ascii (data) VALUES ('[Invalid UTF-8]  Blob: \xf4\xa5\xa3\xa5');"

I then right-clicked the table in the treeview, and selected the option to
view all rows, and immediately saw an error:

2018-06-05 12:23:27,319: SQL pgadmin: Execute (async) for server #1 -
CONN:1187535 (Query-id: 8522474):
SELECT * FROM public.sql_ascii
ORDER BY id ASC
2018-06-05 12:23:27,320: ERROR pgadmin: Failed to execute query
(execute_async) for the server #1 - CONN:1187535(Query-id: 8522474):
Error Message:ERROR:  invalid byte sequence for encoding "UTF8": 0x80
SQL state: 22021

Running "SELECT * FROM sql_ascii" in the query tool resulted in the same
error, however, if I ran "SET client_encoding = 'SQL_ASCII';" first, I do
see results.

I have confirmed that I've restarted the server after applying the patch.

What am I missing? Why don't we just set the client_encoding to SQL_ASCII
if it's a SQL_ASCII database?

Note that this testing was on Python 2.7.10 on MacOS.

> On Tue, Jun 5, 2018 at 10:15 AM, Aditya Toshniwal <aditya.toshniwal@
> enterprisedb.com> wrote:
>
>> Hi
>>

---

From: Aditya Toshniwal <[email-masked]>
Date: Tue, 5 Jun 2018 17:51:00 +0530

Hi Dave,

On Tue, Jun 5, 2018 at 4:56 PM, Dave Page <[email-masked]> wrote:

> Hi
>
> On Tue, Jun 5, 2018 at 9:50 AM, Aditya Toshniwal <aditya.toshniwal@
> enterprisedb.com> wrote:
>
>> Hi Hackers,
>>
>> The test cases were running fine when the module was specified using
>> --pkg but were failing in complete run. Fixed that.
>>
>

It is by default same as the server encoding. But, the following existing
code in  web/pgadmin/utils/driver/psycopg2/connection.py makes the
client_encoding as UNICODE for every connection. I am not sure it should be
removed.

        status = _execute(cur, "SET DateStyle=ISO;"

                               "SET client_min_messages=notice;"

                               "SET bytea_output=escape;"

                               "SET client_encoding='UNICODE';")

>>>>> The below code is minor fix related to connection status of sql
>>>>> editor. Can you please share the code snippet if it is not the below.
>>>>>

---

From: Dave Page <[email-masked]>
Date: Tue, 5 Jun 2018 13:35:24 +0100

Hi

On Tue, Jun 5, 2018 at 1:21 PM, Aditya Toshniwal <
[email-masked]> wrote:

>> /Library/PostgreSQL/9.4/bin/psql -d sql_ascii -U postgres -c "INSERT
>> INTO sql_acsii (data) VALUES ('[Windows-1252]   Euro: \x80   Double dagger:
>> \x87');"
>> /Library/PostgreSQL/9.4/bin/psql -d sql_ascii -U postgres -c "INSERT
>> INTO sql_ascii (data) VALUES ('[Latin-1]   Yen: \xa5   Half: \xbd');"
>> /Library/PostgreSQL/9.4/bin/psql -d sql_ascii -U postgres -c "INSERT
>> INTO sql_ascii (data) VALUES ('[Japanese]   Ship: \xe8\x88\xb9');"
>> /Library/PostgreSQL/9.4/bin/psql -d sql_ascii -U postgres -c "INSERT
>> INTO sql_ascii (data) VALUES ('[Invalid UTF-8]  Blob: \xf4\xa5\xa3\xa5');"
>>
>> I then right-clicked the table in the treeview, and selected the option
>> to view all rows, and immediately saw an error:
>>

It was probably before you joined, but I have said a number of times that
pgAdmin 3 handled this differently and that maybe we should do it the same
way here. See
https://git.postgresql.org/gitweb/?p=pgadmin3.git;a=blob;f=pgadmin/db/pgConn.cpp,
in the pgConn::Initialize() function.

Either way, your patch isn't working for me.

---

From: Aditya Toshniwal <[email-masked]>
Date: Tue, 5 Jun 2018 18:19:21 +0530

Hi Dave,

The problem of SQL ASCII is solved with the patch, and not related to
setting the client encoding of the sql window. I can see there is no SET
call in pgAdmin3 for client_encoding.  I can remove the SET
client_encoding='UNICODE'; that will solve the problem. But, can you please
let me know why that was added.

Will remove the set call and will send you the updated patch if everything
works fine.

Thanks and Regards,
Aditya Toshniwal

On Tue, Jun 5, 2018 at 6:05 PM, Dave Page <[email-masked]> wrote:

> Hi
>
> On Tue, Jun 5, 2018 at 1:21 PM, Aditya Toshniwal <aditya.toshniwal@
> enterprisedb.com> wrote:
>
>> Hi Dave,
>>
>>
>>> What am I missing? Why don't we just set the client_encoding to
>>> SQL_ASCII if it's a SQL_ASCII database?
>>>
>>
> way here. See https://git.postgresql.org/gitweb/?p=pgadmin3.git;a=
> blob;f=pgadmin/db/pgConn.cpp, in the pgConn::Initialize() function.
>

---

From: Aditya Toshniwal <[email-masked]>
Date: Tue, 5 Jun 2018 18:23:35 +0530

Also, there are limitations in psycopg2 as well related to encoding, which
makes it even more difficult to make behaviour similar to pgAdmin3 as there
are encodings and decodings done at pyscopg2 end also.

Thanks and Regards,
Aditya Toshniwal

On Tue, Jun 5, 2018 at 6:19 PM, Aditya Toshniwal <
[email-masked]> wrote:

> client_encoding='UNICODE'; that will solve the problem. But, can you
> please let me know why that was added.
>
>>>> Running "SELECT * FROM sql_ascii" in the query tool resulted in the
>>>> same error, however, if I ran "SET client_encoding = 'SQL_ASCII';" first, I
>>>> do see results.
>>>>
>>> It is by default same as the server encoding. But, the following
>>> existing code in  web/pgadmin/utils/driver/psycopg2/connection.py makes
>>> the client_encoding as UNICODE for every connection. I am not sure it
>>> should be removed.
>>>
>> way here. See https://git.postgresql.org/gitweb/?p=pgadmin3.git;a=blob
>> ;f=pgadmin/db/pgConn.cpp, in the pgConn::Initialize() function.
>>

---

From: Dave Page <[email-masked]>
Date: Tue, 5 Jun 2018 13:55:59 +0100

On Tue, Jun 5, 2018 at 1:49 PM, Aditya Toshniwal <
[email-masked]> wrote:

> setting the client encoding of the sql window.
>

No it's not. It doesn't work for me as I said (and showed the example of).

> I can see there is no SET call in pgAdmin3 for client_encoding.  I can
> remove the SET client_encoding='UNICODE'; that will solve the problem.
> But, can you please let me know why that was added.
>

There is, but it's inside an API call (PQsetClientEncoding):

300
<https://git.postgresql.org/gitweb/?p=pgadmin3.git;a=blob;f=pgadmin/db/pgConn.cpp#l300>
            wxLogInfo(wxT("Setting client_encoding to '%s'"), encoding.c_str());
301
<https://git.postgresql.org/gitweb/?p=pgadmin3.git;a=blob;f=pgadmin/db/pgConn.cpp#l301>
            if (PQsetClientEncoding(conn, encoding.ToAscii()))
302
<https://git.postgresql.org/gitweb/?p=pgadmin3.git;a=blob;f=pgadmin/db/pgConn.cpp#l302>
            {
303
<https://git.postgresql.org/gitweb/?p=pgadmin3.git;a=blob;f=pgadmin/db/pgConn.cpp#l303>
                wxLogError(wxT("%s"), GetLastError().c_str());
304
<https://git.postgresql.org/gitweb/?p=pgadmin3.git;a=blob;f=pgadmin/db/pgConn.cpp#l304>
            }
305
<https://git.postgresql.org/gitweb/?p=pgadmin3.git;a=blob;f=pgadmin/db/pgConn.cpp#l305>

> Will remove the set call and will send you the updated patch if everything

No, we need to ensure the client encoding is set correctly. It just needs
to be set to SQL_ASCII if it's a SQL_ASCII database (I believe).

---

From: Aditya Toshniwal <[email-masked]>
Date: Tue, 5 Jun 2018 18:33:21 +0530

Hi

On Tue, Jun 5, 2018 at 6:25 PM, Dave Page <[email-masked]> wrote:

>
>
> On Tue, Jun 5, 2018 at 1:49 PM, Aditya Toshniwal <aditya.toshniwal@
> enterprisedb.com> wrote:
>
>> Hi Dave,
>>

After setting the client_encoding to SQL_ASCII you got the output.
Previously, it used to fail in the back end itself because python encoding
failure. That is fixed.
The error ERROR:  invalid byte sequence for encoding "UTF8": 0x80 is thrown
by postgres and not python or pgAdmin4. You will get the same error even if
you
connect from psql.

> Oops ! Missed that. Apologies.

>
>
>>
>> Will remove the set call and will send you the updated patch if
>> everything works fine.
>>
>
Need to rework on the initialise method. Will come with an updated. patch.
Sorry for trouble.

>>>>>> PFA updated patch. The sqleditor change is sent separately and
>>>>>> removed from current patch as suggested.
>>>>> I did a quick test by creating a SQL_ASCII database containing a
>>>>> simple table:
>>>>>
>>>>> I then right-clicked the table in the treeview, and selected the
>>>>> option to view all rows, and immediately saw an error:
>>>>>
>>> It was probably before you joined, but I have said a number of times
>>> that pgAdmin 3 handled this differently and that maybe we should do it the
>>> same way here. See https://git.postgresql.org
>>> /gitweb/?p=pgadmin3.git;a=blob;f=pgadmin/db/pgConn.cpp, in the
>>> pgConn::Initialize() function.
>>>

---

From: Dave Page <[email-masked]>
Date: Tue, 5 Jun 2018 14:12:37 +0100

Hi

On Tue, Jun 5, 2018 at 2:03 PM, Aditya Toshniwal <
[email-masked]> wrote:

> The error ERROR:  invalid byte sequence for encoding "UTF8": 0x80 is
> thrown by postgres and not python or pgAdmin4. You will get the same error
> even if you
> connect from psql.
>

Sure - but that is not a fix. You have no way of running the SET command if
you're using "view data" - and in the query tool, users just expect it to
work (as it did in pgAdmin 3).

>>>>>> 2018-06-05 12:23:27,319: SQL pgadmin: Execute (async) for server #1
>>>>>> - CONN:1187535 (Query-id: 8522474):

---

From: Aditya Toshniwal <[email-masked]>
Date: Wed, 6 Jun 2018 18:32:55 +0530

Hi Hackers,

PFA updated patch as the previous one was not working as expected. I have
tried to make it similar to that of pgAdmin3 and you do not need to change
client_encoding as it is set now based on server encoding. It works fine
with "view data" also.
The only problem is, I cannot find equivalent codec for wxConvLibc in
python. The closest one I could find is raw_unicode_escape. So, in a
SQL_ASCII database, non ASCII characters may differ in pgAdmin4 and
pgAdmin3, but it will display results.

Dave,
You need to add "E" before the string to be inserted, otherwise \x will be
considered as a plain string.
INSERT INTO sql_ascii (data) VALUES (E'[Invalid UTF-8]  Blob:
\xf4\xa5\xa3\xa5');

Kindly review.

Thanks and Regards,
Aditya Toshniwal

On Tue, Jun 5, 2018 at 6:42 PM, Dave Page <[email-masked]> wrote:

> Hi
>
> On Tue, Jun 5, 2018 at 2:03 PM, Aditya Toshniwal <aditya.toshniwal@
> enterprisedb.com> wrote:
>
>> Hi
>>
> Sure - but that is not a fix. You have no way of running the SET command
> if you're using "view data" - and in the query tool, users just expect it
> to work (as it did in pgAdmin 3).
>
>
>>
>>>
>>> No, we need to ensure the client encoding is set correctly. It just
>>> needs to be set to SQL_ASCII if it's a SQL_ASCII database (I believe).
>>>
>>>
>> Need to rework on the initialise method. Will come with an updated.
>> patch. Sorry for trouble.
>>
>>>
>>>>
>>>>>>>> The test cases were running fine when the module was specified
>>>>>>>> using --pkg but were failing in complete run. Fixed that.
>>>>>>>>
>>>>>>>
>>>>>>> /Library/PostgreSQL/9.4/bin/psql -d sql_ascii -U postgres -c
>>>>>>> "INSERT INTO sql_acsii (data) VALUES ('[Windows-1252]   Euro: \x80   Double
>>>>>>> dagger: \x87');"
>>>>>>> /Library/PostgreSQL/9.4/bin/psql -d sql_ascii -U postgres -c
>>>>>>> "INSERT INTO sql_ascii (data) VALUES ('[Latin-1]   Yen: \xa5   Half:
>>>>>>> \xbd');"
>>>>>>> /Library/PostgreSQL/9.4/bin/psql -d sql_ascii -U postgres -c
>>>>>>> "INSERT INTO sql_ascii (data) VALUES ('[Japanese]   Ship: \xe8\x88\xb9');"
>>>>>>> /Library/PostgreSQL/9.4/bin/psql -d sql_ascii -U postgres -c
>>>>>>> "INSERT INTO sql_ascii (data) VALUES ('[Invalid UTF-8]  Blob:
>>>>>>> \xf4\xa5\xa3\xa5');"
>>>>>>>
>>>>>>>>> No its not. It is something I found when I was working on
>>>>>>>>> SQL_ASCII related changes.
diff --git a/web/pgadmin/tools/sqleditor/tests/test_encoding_charset.py b/web/pgadmin/tools/sqleditor/tests/test_encoding_charset.py
new file mode 100644
index 0000000..0826dec
--- /dev/null
+++ b/web/pgadmin/tools/sqleditor/tests/test_encoding_charset.py
@@ -0,0 +1,103 @@
+##########################################################################
+#
+# pgAdmin 4 - PostgreSQL Tools
+#
+# Copyright (C) 2013 - 2018, The pgAdmin Development Team
+# This software is released under the PostgreSQL Licence
+#
+##########################################################################
+
+from pgadmin.utils.route import BaseTestGenerator
+from pgadmin.browser.server_groups.servers.databases.tests import utils as \
+    database_utils
+from regression import parent_node_dict
+from regression.python_test_utils import test_utils
+import json
+
+
+class TestEncodingCharset(BaseTestGenerator):
+    """
+    This class validates character support in pgAdmin4 for
+    different PostgresDB encodings
+    """
+    scenarios = [
+        (
+            'With Encoding UTF8',
+            dict(
+                db_encoding='UTF8',
+                lc_collate='C',
+                test_str='A'
+            )),
+        (
+            'With Encoding WIN1252',
+            dict(
+                db_encoding='WIN1252',
+                lc_collate='C',
+                test_str='A'
+            )),
+        (
+            'With Encoding EUC_CN',
+            dict(
+                db_encoding='EUC_CN',
+                lc_collate='C',
+                test_str='A'
+            )),
+        (
+            'With Encoding SQL_ASCII',
+            dict(
+                db_encoding='SQL_ASCII',
+                lc_collate='C',
+                test_str='\\255'
+            )),
+    ]
+
+    def setUp(self):
+        self.encode_db_name = 'encoding_' + self.db_encoding
+        self.encode_sid = self.server_information['server_id']
+        self.encode_did = test_utils.create_database(
+            self.server, self.encode_db_name,
+            (self.db_encoding, self.lc_collate))
+
+    def runTest(self):
+
+        db_con = database_utils.connect_database(self,
+                                                 test_utils.SERVER_GROUP,
+                                                 self.encode_sid,
+                                                 self.encode_did)
+        if not db_con["info"] == "Database connected.":
+            raise Exception("Could not connect to the database.")
+
+        # Initialize query tool
+        url = '/datagrid/initialize/query_tool/{0}/{1}/{2}'.format(
+            test_utils.SERVER_GROUP, self.encode_sid, self.encode_did)
+        response = self.tester.post(url)
+        self.assertEquals(response.status_code, 200)
+
+        response_data = json.loads(response.data.decode('utf-8'))
+        self.trans_id = response_data['data']['gridTransId']
+
+        # Check character
+        url = "/sqleditor/query_tool/start/{0}".format(self.trans_id)
+        sql = "select E'{0}';".format(self.test_str)
+        response = self.tester.post(url, data=json.dumps({"sql": sql}),
+                                    content_type='html/json')
+        self.assertEquals(response.status_code, 200)
+        url = '/sqleditor/poll/{0}'.format(self.trans_id)
+        response = self.tester.get(url)
+        self.assertEquals(response.status_code, 200)
+        response_data = json.loads(response.data.decode('utf-8'))
+        self.assertEquals(response_data['data']['rows_fetched_to'], 1)
+
+        database_utils.disconnect_database(self, self.encode_sid,
+                                           self.encode_did)
+
+    def tearDown(self):
+        main_conn = test_utils.get_db_connection(
+            self.server['db'],
+            self.server['username'],
+            self.server['db_password'],
+            self.server['host'],
+            self.server['port'],
+            self.server['sslmode']
+        )
+        test_utils.drop_database(main_conn, self.encode_db_name)
diff --git a/web/pgadmin/utils/driver/psycopg2/connection.py b/web/pgadmin/utils/driver/psycopg2/connection.py
index cfd161a..e2538d1 100644
--- a/web/pgadmin/utils/driver/psycopg2/connection.py
+++ b/web/pgadmin/utils/driver/psycopg2/connection.py
@@ -33,7 +33,7 @@ from pgadmin.utils import get_complete_file_path
 from ..abstract import BaseDriver, BaseConnection
 from .cursor import DictCursor
 from .typecast import register_global_typecasters, \
-    register_string_typecasters, register_binary_typecasters, \
+    register_binary_typecasters, \
     register_array_to_string_typecasters, ALL_JSON_TYPES

@@ -387,8 +387,6 @@ class Connection(BaseConnection):
             else:
                 self.conn.autocommit = True
 
-        register_string_typecasters(self.conn)
-
         if self.array_to_string:
             register_array_to_string_typecasters(self.conn)
 
@@ -397,10 +395,19 @@ class Connection(BaseConnection):
         if self.use_binary_placeholder:
             register_binary_typecasters(self.conn)
 
-        status = _execute(cur, "SET DateStyle=ISO;"
-                               "SET client_min_messages=notice;"
-                               "SET bytea_output=escape;"
-                               "SET client_encoding='UNICODE';")
+        if self.conn.encoding not in ('SQL_ASCII', 'SQLASCII',
+                                      'MULE_INTERNAL', 'MULEINTERNAL'):
+            status = _execute(cur, "SET DateStyle=ISO;"
+                                   "SET client_min_messages=notice;"
+                                   "SET bytea_output=escape;"
+                                   "SET client_encoding='UNICODE';")
+
+            encodings[self.conn.encoding] = 'utf-8'
+        else:
+            status = _execute(cur, "SET DateStyle=ISO;"
+                                   "SET client_min_messages=notice;"
+                                   "SET bytea_output=escape;")
+            encodings[self.conn.encoding] = 'raw_unicode_escape'
 
         if status is not None:
             self.conn.close()
diff --git a/web/pgadmin/utils/driver/psycopg2/typecast.py b/web/pgadmin/utils/driver/psycopg2/typecast.py
index f136604..a8f6c38 100644
--- a/web/pgadmin/utils/driver/psycopg2/typecast.py
+++ b/web/pgadmin/utils/driver/psycopg2/typecast.py
@@ -163,49 +163,6 @@ def register_global_typecasters():
     psycopg2.extensions.register_type(pg_array_types_to_array_of_string_type)

-def register_string_typecasters(connection):
-    if connection.encoding != 'UTF8':
-        # In python3 when database encoding is other than utf-8 and client
-        # encoding is set to UNICODE then we need to map data from database
-        # encoding to utf-8.
-        # This is required because when client encoding is set to UNICODE then
-        # psycopg assumes database encoding utf-8 and not the actual encoding.
-        # Not sure whether it's bug or feature in psycopg for python3.
-        if sys.version_info >= (3,):
-            def return_as_unicode(value, cursor):
-                if value is None:
-                    return None
-                # Treat value as byte sequence of database encoding and then
-                # decode it as utf-8 to get correct unicode value.
-                return bytes(
-                    value, encodings[cursor.connection.encoding]
-                ).decode('utf-8')
-
-            unicode_type = psycopg2.extensions.new_type(
-                # "char", name, text, character, character varying
-                (19, 18, 25, 1042, 1043, 0),
-                'UNICODE', return_as_unicode)
-        else:
-            def return_as_unicode(value, cursor):
-                if value is None:
-                    return None
-                # Decode it as utf-8 to get correct unicode value.
-                return value.decode('utf-8')
-
-            unicode_type = psycopg2.extensions.new_type(
-                # "char", name, text, character, character varying
-                (19, 18, 25, 1042, 1043, 0),
-                'UNICODE', return_as_unicode)
-
-        unicode_array_type = psycopg2.extensions.new_array_type(
-            # "char"[], name[], text[], character[], character varying[]
-            (1002, 1003, 1009, 1014, 1015, 0
-             ), 'UNICODEARRAY', unicode_type)
-
-        psycopg2.extensions.register_type(unicode_type)
-        psycopg2.extensions.register_type(unicode_array_type)
-
-
 def register_binary_typecasters(connection):
     psycopg2.extensions.register_type(
         psycopg2.extensions.new_type(
diff --git a/web/regression/python_test_utils/test_utils.py b/web/regression/python_test_utils/test_utils.py
index 3e517b6..464a09e 100644
--- a/web/regression/python_test_utils/test_utils.py
+++ b/web/regression/python_test_utils/test_utils.py
@@ -116,7 +116,7 @@ def clear_node_info_dict():
         del node_info_dict[node][:]

---

From: Dave Page <[email-masked]>
Date: Thu, 7 Jun 2018 11:37:28 +0100

Hi

On Wed, Jun 6, 2018 at 2:02 PM, Aditya Toshniwal <aditya.toshniwal@

> client_encoding as it is set now based on server encoding. It works fine

- In connection.py, at ~409, shouldn't we set the client_encoding to
SQL_ASCII? Otherwise it could be overridden with something unexpected if
the client has PGCLIENTENCODING set for example.

- With or without that change, I get the following test failure on macOS
with Python 2.7.10:

======================================================================
ERROR: runTest
(pgadmin.tools.sqleditor.tests.test_encoding_charset.TestEncodingCharset)
With Encoding SQL_ASCII
----------------------------------------------------------------------
Traceback (most recent call last):
  File
"/Users/dpage/git/pgadmin4/web/pgadmin/tools/sqleditor/tests/test_encoding_charset.py",
line 86, in runTest
    response = self.tester.get(url)
  File
"/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/werkzeug/test.py",
line 830, in get
    return self.open(*args, **kw)
  File
"/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/testing.py",
line 127, in open
    follow_redirects=follow_redirects)
  File
"/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/werkzeug/test.py",
line 803, in open
    response = self.run_wsgi_app(environ, buffered=buffered)
  File
"/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/werkzeug/test.py",
line 716, in run_wsgi_app
    rv = run_wsgi_app(self.application, environ, buffered=buffered)
  File
"/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/werkzeug/test.py",
line 923, in run_wsgi_app
    app_rv = app(environ, start_response)
  File
"/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
line 1997, in __call__
    return self.wsgi_app(environ, start_response)
  File
"/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
line 1985, in wsgi_app
    response = self.handle_exception(e)
  File
"/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
line 1540, in handle_exception
    reraise(exc_type, exc_value, tb)
  File
"/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
line 1982, in wsgi_app
    response = self.full_dispatch_request()
  File
"/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
line 1614, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File
"/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
line 1517, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File
"/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
line 1612, in full_dispatch_request
    rv = self.dispatch_request()
  File
"/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
line 1598, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File
"/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask_login.py",
line 792, in decorated_view
    return func(*args, **kwargs)
  File "/Users/dpage/git/pgadmin4/web/pgadmin/tools/sqleditor/__init__.py",
line 576, in poll
    'oids': oids
  File "/Users/dpage/git/pgadmin4/web/pgadmin/utils/ajax.py", line 61, in
make_json_response
    separators=(',', ':')),
  File
"/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/simplejson/__init__.py",
line 399, in dumps
    **kw).encode(obj)
  File
"/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/simplejson/encoder.py",
line 291, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File
"/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/simplejson/encoder.py",
line 373, in iterencode
    return _iterencode(o, 0)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xad in position 0:
invalid start byte

----------------------------------------------------------------------
Ran 317 tests in 30.692s

FAILED (errors=1, skipped=21)

> pgAdmin3, but it will display results.

Yeah, I think that's fine. For the small number of people with SQL_ASCII
databases, seeing escaped characters is better than nothing.

> INSERT INTO sql_ascii (data) VALUES (E'[Invalid UTF-8]  Blob:

Yeah, sorry - I copied the wrong version of the query :-(

>>>>> remove the SET client_encoding='UNICODE'; that will solve the
>>>>> problem. But, can you please let me know why that was added.
>>>>>
>>>>
>>>>>>>> 2018-06-05 12:23:27,319: SQL pgadmin: Execute (async) for server
>>>>>>>> #1 - CONN:1187535 (Query-id: 8522474):

---

From: Aditya Toshniwal <[email-masked]>
Date: Thu, 7 Jun 2018 16:35:46 +0530

Hi Dave,

On Thu, Jun 7, 2018 at 4:07 PM, Dave Page <[email-masked]> wrote:

> Hi
>
> On Wed, Jun 6, 2018 at 2:02 PM, Aditya Toshniwal <
Yeah I agree, it would be better to add. Will add the change.

> - With or without that change, I get the following test failure on macOS
It works fine on my machine with Python 2.7 and macOS. Could you please let
me know the Postgres DB version also.
Will test on few more machines.

> ERROR: runTest (pgadmin.tools.sqleditor.tests.test_encoding_charset.
> TestEncodingCharset)
> With Encoding SQL_ASCII
>   File "/Users/dpage/git/pgadmin4/web/pgadmin/tools/sqleditor/
> tests/test_encoding_charset.py", line 86, in runTest
>   File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/werkzeug/test.py",
> line 830, in get
>   File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/testing.py",
> line 127, in open
>   File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/werkzeug/test.py",
> line 803, in open
>   File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/werkzeug/test.py",
>   File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/werkzeug/test.py",
>   File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
> line 1997, in __call__
>   File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
> line 1985, in wsgi_app
>   File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
>   File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
> line 1982, in wsgi_app
>   File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
>   File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
>   File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
>   File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
>   File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask_login.py",
>   File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-
> packages/simplejson/__init__.py", line 399, in dumps
>     **kw).encode(obj)
>   File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-
> packages/simplejson/encoder.py", line 291, in encode
>   File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-
> packages/simplejson/encoder.py", line 373, in iterencode
>     return _iterencode(o, 0)
>> You need to add "E" before the string to be inserted, otherwise \x will
>> be considered as a plain string.
>>>>>> I can see there is no SET call in pgAdmin3 for client_encoding.  I
>>>>>> can remove the SET client_encoding='UNICODE'; that will solve the
>>>>>>>>> Running "SELECT * FROM sql_ascii" in the query tool resulted in
>>>>>>>>> the same error, however, if I ran "SET client_encoding = 'SQL_ASCII';"
>>>>>>>>> first, I do see results.
>>>>>>>>>
>>>>>>>>>>>>> The below code is minor fix related to connection status of
>>>>>>>>>>>>> sql editor. Can you please share the code snippet if it is not the below.
>>>>>>>>>>>>>

---

From: Dave Page <[email-masked]>
Date: Thu, 7 Jun 2018 12:11:16 +0100

Hi

On Thu, Jun 7, 2018 at 12:05 PM, Aditya Toshniwal <
[email-masked]> wrote:

>>> PFA updated patch as the previous one was not working as expected. I
>>> have tried to make it similar to that of pgAdmin3 and you do not need to
>>> change client_encoding as it is set now based on server encoding. It works
>>> fine with "view data" also.
>>>
>>
> It works fine on my machine with Python 2.7 and macOS. Could you please
> let me know the Postgres DB version also.
>

PostgreSQL 9.4.10 on x86_64-apple-darwin, compiled by
i686-apple-darwin11-llvm-gcc-4.2 (GCC) 4.2.1 (Based on Apple Inc. build
5658) (LLVM build 2336.11.00), 64-bit

>> ERROR: runTest (pgadmin.tools.sqleditor.tests
>> .test_encoding_charset.TestEncodingCharset)
>> With Encoding SQL_ASCII
>>   File "/Users/dpage/git/pgadmin4/web/pgadmin/tools/sqleditor/tests/test_encoding_charset.py",
>> line 86, in runTest
>>   File "/Users/dpage/git/pgadmin4/web/pgadmin/utils/ajax.py", line 61,
>> in make_json_response
>>     separators=(',', ':')),
>>   File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/simplejson/__init__.py",
>> line 399, in dumps
>>     **kw).encode(obj)
>>   File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/simplejson/encoder.py",
>> line 291, in encode
>>   File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/simplejson/encoder.py",
>> line 373, in iterencode
>>     return _iterencode(o, 0)
>>>>>>> The problem of SQL ASCII is solved with the patch, and not related
>>>>>>> to setting the client encoding of the sql window.
>>>>>>>
>>>>>>
>>>> Sure - but that is not a fix. You have no way of running the SET
>>>> command if you're using "view data" - and in the query tool, users just
>>>> expect it to work (as it did in pgAdmin 3).
>>>>
>>>>
>>>>>
>>>>>>
>>>>>>>> It was probably before you joined, but I have said a number of
>>>>>>>> times that pgAdmin 3 handled this differently and that maybe we should do
>>>>>>>> it the same way here. See https://git.postgresql.org

---

From: Aditya Toshniwal <[email-masked]>
Date: Thu, 14 Jun 2018 11:34:47 +0530

Hi Hackers,

PFA updated patch.

On Thu, Jun 7, 2018 at 4:41 PM, Dave Page <[email-masked]> wrote:

> Hi
>
> On Thu, Jun 7, 2018 at 12:05 PM, Aditya Toshniwal <aditya.toshniwal@
> enterprisedb.com> wrote:
>
>> Hi Dave,
>>
>>
This is fixed. There was a problem with json dumps. Now, json dumps will be
done based on connection encoding.

>>>>>>>>>>> 2018-06-05 12:23:27,319: SQL pgadmin: Execute (async) for
>>>>>>>>>>> server #1 - CONN:1187535 (Query-id: 8522474):
>>>>>>>>>> existing code in  web/pgadmin/utils/driver/psycopg2/connection.py
>>>>>>>>>> makes the client_encoding as UNICODE for every connection. I am
>>>>>>>>>> not sure it should be removed.
>>>>>>>>>>

---

From: Aditya Toshniwal <[email-masked]>
Date: Thu, 14 Jun 2018 11:35:35 +0530

I am sorry I missed the attachment. :(
PFA.

On Thu, Jun 14, 2018 at 11:34 AM, Aditya Toshniwal <
[email-masked]> wrote:

>>>> - With or without that change, I get the following test failure on
>>>> macOS with Python 2.7.10:
>>>>
> This is fixed. There was a problem with json dumps. Now, json dumps will
> be done based on connection encoding.
>
>>
>>>>
>>>>
>>>> Yeah, I think that's fine. For the small number of people with
>>>> SQL_ASCII databases, seeing escaped characters is better than nothing.
>>>>
>>>>
>>>>>
>>>>>
>>>>> Dave,
>>>>> You need to add "E" before the string to be inserted, otherwise \x
>>>>> will be considered as a plain string.
>>>>>>>>>>>> I did a quick test by creating a SQL_ASCII database containing
>>>>>>>>>>>> a simple table:
>>>>>>>>>>>>
>>>>>>>>>>>> I then right-clicked the table in the treeview, and selected
>>>>>>>>>>>> the option to view all rows, and immediately saw an error:
>>>>>>>>>>>>
>>>>>>>>>>>> 2018-06-05 12:23:27,320: ERROR pgadmin: Failed to execute
>>>>>>>>>>>> query (execute_async) for the server #1 - CONN:1187535(Query-id: 8522474):

---

From: Dave Page <[email-masked]>
Date: Fri, 15 Jun 2018 09:38:19 +0100

Hi

> I am sorry I missed the attachment. :(

It looks like the encoding names are getting munged somewhere. I see you've
accounted for that to some degree in connection.py (you have both
SQL_ASCII/MULE_INTERNAL and SQLASCII/MULEINTERNAL), however it doesn't seem
to be enough as I'm getting the following error when trying to download CSV
from the query tool. Can we ensure that conn.encoding contains an un-munged
value at all times, or is that coming from psycopg2?

2018-06-15 09:32:28,799: INFO werkzeug: 127.0.0.1 - - [15/Jun/2018
09:32:28] "GET
/sqleditor/query_tool/download/2732923?query=SELECT%20*%20FROM%20public.sql_ascii%0AORDER%20BY%20id%20ASC%20&filename=sql_ascii.csv
HTTP/1.1" 500 -
2018-06-15 09:32:28,801: ERROR werkzeug: Error on request:
Traceback (most recent call last):
  File
"/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/werkzeug/serving.py",
line 270, in run_wsgi
    execute(self.server.app)
  File
"/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/werkzeug/serving.py",
line 260, in execute
    for data in application_iter:
  File
"/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/werkzeug/wsgi.py",
line 870, in __next__
    return self._next()
  File
"/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/werkzeug/wrappers.py",
line 82, in _iter_encoded
    for item in iterable:
  File
"/Users/dpage/git/pgadmin4/web/pgadmin/utils/driver/psycopg2/connection.py",
line 752, in gen
    column_name = column_name.decode(conn_encoding)
LookupError: unknown encoding: SQLASCII

---

From: Aditya Toshniwal <[email-masked]>
Date: Fri, 15 Jun 2018 14:33:31 +0530

Hi Dave,

On Fri, Jun 15, 2018 at 2:08 PM, Dave Page <[email-masked]> wrote:

> Hi
>
> On Thu, Jun 14, 2018 at 7:05 AM, Aditya Toshniwal <aditya.toshniwal@
> enterprisedb.com> wrote:
>
> It looks like the encoding names are getting munged somewhere. I see
> you've accounted for that to some degree in connection.py (you have both
​That is done by pyscopg2 and conn.encoding is a psycopg2 connection
property.​

> 09:32:28] "GET /sqleditor/query_tool/download/2732923?query=SELECT%
> 20*%20FROM%20public.sql_ascii%0AORDER%20BY%20id%20ASC%20&filename=sql_ascii.csv
> HTTP/1.1" 500 -
>   File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/werkzeug/serving.py",
> line 270, in run_wsgi
>     execute(self.server.app)
>   File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/werkzeug/serving.py",
> line 260, in execute
>   File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/werkzeug/wsgi.py",
> line 870, in __next__
>     return self._next()
> packages/werkzeug/wrappers.py", line 82, in _iter_encoded
>     for item in iterable:
>   File "/Users/dpage/git/pgadmin4/web/pgadmin/utils/driver/psycopg2/connection.py",
> line 752, in gen

​This is because there is code bug here. Below is code used to decode a
column name. Connection encoding and python encoding are two different
things. Python does not know what SQLASCII is. This will work with UTF-8
because python has decoder with same name. I tried to download CSV with the
original code without changes and it fails there too. I will fix this and
will send the updated patch. I should have checked this.
conn_encoding = cur.connection.encoding
column_name = column_name.decode(conn_encoding)​

---

From: Aditya Toshniwal <[email-masked]>
Date: Mon, 18 Jun 2018 18:44:11 +0530

Hi Hackers,

Attached is the updated patch which includes the fix for Download CSV fail
in SQL_ASCII database, which is RM3250
<https://redmine.postgresql.org/issues/3250>
This should fix RM3289 and RM3250. As they interrelated, sending the
combined patch.
Kindly review.

On Fri, Jun 15, 2018 at 2:33 PM, Aditya Toshniwal <
[email-masked]> wrote:

>> 09:32:28] "GET /sqleditor/query_tool/download/2732923?query=SELECT%20*%
>> 20FROM%20public.sql_ascii%0AORDER%20BY%20id%20ASC%20&filename=sql_ascii.csv
>> HTTP/1.1" 500 -
>>   File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/werkzeug/wrappers.py",

---

From: Dave Page <[email-masked]>
Date: Mon, 18 Jun 2018 21:53:35 +0100

Hi

> This should fix RM3289 and RM3250. As they interrelated, sending the

This is definitely looking better - both view and save now work as
expected. However, using the test data the I posted upthread, if I try to
edit a value (in this case by adding a couple of chars to the end of the
data in row 2) I get:

2018-06-18 16:41:40,895: SQL pgadmin: Execute (void) for server #1 -
DB:ascii (Query-id: 3093186):
UPDATE public.ascii SET
data = %(data)s::text WHERE
id = '2';
2018-06-18 16:41:41,027: INFO werkzeug: 127.0.0.1 - - [18/Jun/2018
16:41:41] "POST /sqleditor/save/2805058 HTTP/1.1" 500 -
2018-06-18 16:41:41,042: ERROR werkzeug: Error on request:
Traceback (most recent call last):
  File
"/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/werkzeug/serving.py",
line 270, in run_wsgi
    execute(self.server.app)
  File
"/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/werkzeug/serving.py",
line 258, in execute
    application_iter = app(environ, start_response)
  File
"/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
line 1997, in __call__
    return self.wsgi_app(environ, start_response)
  File
"/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
line 1985, in wsgi_app
    response = self.handle_exception(e)
  File
"/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
line 1540, in handle_exception
    reraise(exc_type, exc_value, tb)
  File
"/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
line 1982, in wsgi_app
    response = self.full_dispatch_request()
  File
"/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
line 1614, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File
"/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
line 1517, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File
"/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
line 1612, in full_dispatch_request
    rv = self.dispatch_request()
  File
"/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
line 1598, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File
"/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask_login.py",
line 792, in decorated_view
    return func(*args, **kwargs)
  File "/Users/dpage/git/pgadmin4/web/pgadmin/tools/sqleditor/__init__.py",
line 776, in save
    default_conn)
  File "/Users/dpage/git/pgadmin4/web/pgadmin/tools/sqleditor/command.py",
line 829, in save
    item['sql'], item['data'])
  File
"/Users/dpage/git/pgadmin4/web/pgadmin/utils/driver/psycopg2/connection.py",
line 975, in execute_void
    self.__internal_blocking_execute(cur, query, params)
  File
"/Users/dpage/git/pgadmin4/web/pgadmin/utils/driver/psycopg2/connection.py",
line 629, in __internal_blocking_execute
    cur.execute(query, params)
  File
"/Users/dpage/git/pgadmin4/web/pgadmin/utils/driver/psycopg2/cursor.py",
line 176, in execute
    return _cursor.execute(self, query, params)
UnicodeEncodeError: 'ascii' codec can't encode characters in position
19-21: ordinal not in range(128)

>
> On Fri, Jun 15, 2018 at 2:33 PM, Aditya Toshniwal <aditya.toshniwal@
> enterprisedb.com> wrote:
>
>> Hi Dave,
>>
>>> 09:32:28] "GET /sqleditor/query_tool/download
>>> /2732923?query=SELECT%20*%20FROM%20public.sql_ascii%0AORDER%
>>> 20BY%20id%20ASC%20&filename=sql_ascii.csv HTTP/1.1" 500 -

---

From: Aditya Toshniwal <[email-masked]>
Date: Wed, 20 Jun 2018 19:47:22 +0530

Hi Dave,

Attached is the updated patch. (Playing with encodings is not at all fun :(
)

On Tue, Jun 19, 2018 at 2:23 AM, Dave Page <[email-masked]> wrote:

> Hi
>
> On Mon, Jun 18, 2018 at 2:14 PM, Aditya Toshniwal <aditya.toshniwal@
> enterprisedb.com> wrote:
>
>> Hi Hackers,
>>
>> Attached is the updated patch which includes the fix for Download CSV
>> fail in SQL_ASCII database, which is RM3250
It should fix the error.

>   File "/Users/dpage/git/pgadmin4/web/pgadmin/utils/driver/psycopg2/cursor.py",
> line 176, in execute

---

From: Dave Page <[email-masked]>
Date: Thu, 21 Jun 2018 13:57:42 +0100

Thanks - patch applied!

On Wed, Jun 20, 2018 at 3:17 PM, Aditya Toshniwal <
[email-masked]> wrote:

> Hi Dave,
>
> Attached is the updated patch. (Playing with encodings is not at all fun
> :( )
>

---

From: Aditya Toshniwal <[email-masked]>
Date: Thu, 21 Jun 2018 21:21:43 +0530

Hi Hackers,

PFA patch to make SQL ASCII related changes compatible with Python 2.6.
Dictionary comprehension is not supported in Python 2.6.

On Thu, Jun 21, 2018 at 6:27 PM, Dave Page <[email-masked]> wrote:

> Thanks - patch applied!
>
> On Wed, Jun 20, 2018 at 3:17 PM, Aditya Toshniwal <aditya.toshniwal@
> enterprisedb.com> wrote:
>
>> Hi Dave,
>>
>>>>> ​This is because there is code bug here. Below is code used to decode
>>>>> a column name. Connection encoding and python encoding are two different

---

From: Dave Page <[email-masked]>
Date: Thu, 21 Jun 2018 22:07:02 +0100

Thanks, applied.