public inbox for [email protected]
help / color / mirror / Atom feedFrom: Aditya Toshniwal <[email protected]>
To: Dave Page <[email protected]>
Cc: Joao De Almeida Pereira <[email protected]>
Cc: pgadmin-hackers <[email protected]>
Subject: Re: [pgAdmin4][RM#3289] Can't query SQL_ASCII database.
Date: Wed, 20 Jun 2018 19:47:22 +0530
Message-ID: <CAM9w-_k7FeHMwcOd1RqkCC64pF2b3=WHXpM14YJZ0b45tWUdBw@mail.gmail.com> (raw)
In-Reply-To: <CA+OCxowxJghshFjXCtOUiRa0qqS93pC4m-bPRhdb7Q3AWe+HnQ@mail.gmail.com>
References: <CAM9w-_ntCUguZp2rVJXPsDvWJbWneAz4-AHJoYB+nUTSYMXc9g@mail.gmail.com>
<CA+OCxozRjJY6RWrXpMLD6-dWnm2LdPs6bS+VmJa403vxVoLwhg@mail.gmail.com>
<CAM9w-_m-cJax16zqBonx-d0-HSy5dNoOzB13HWAad5Xv0Wu=CA@mail.gmail.com>
<CANxYE3+qrGeZYzdYEtzxS8i2P+325V5E+n1xVpyzTwn7vRqRDg@mail.gmail.com>
<CAM9w-_mTe0gW+2cdhMutMWyBsRt7Om3G_Em126Ryntf0nOiHbA@mail.gmail.com>
<CAM9w-_mXdJhuRFn87yAuqWtfrhnRgw1KvU2PpthL-hbnOt2QpA@mail.gmail.com>
<CAE+jjampAPoDq7xJgeNf1L=i_1jf83PLr=dZYicfkEMp6_UBAg@mail.gmail.com>
<CAM9w-_k2Fo-qDy+qxM5i=Aba__TAWxrryR+m3izTXB=koToKcQ@mail.gmail.com>
<CA+OCxoz+Wo3pUuu8SMBzAsQvrOjYLPQHCfQMtO3X3HKezKcapw@mail.gmail.com>
<CAM9w-_nEDycbeEEeUhKwXjxmkNC39YzqhggJR3Frc4pq6WcJ0Q@mail.gmail.com>
<CAM9w-_kDQkAWRZrRRtmMDz8DZ5x2wGoUro6oQbQqTzLmZZGyBw@mail.gmail.com>
<CAE+jjametYGjStNFZFW544Jcm_by1OABtb7yFQWKGzyfk08QiA@mail.gmail.com>
<CAM9w-_=rsazJWCu5xRqLGkYX3FW22vVuyc0NcR5TPskO9PApUg@mail.gmail.com>
<CAE+jjakwbfsbwkvwa6N1QGjGeVb36sWnde+C6amnHQLHdTE3kQ@mail.gmail.com>
<CAM9w-_km5Zv=Gw6CUZF8m3zzacOKL-ir-QNw3xPjP2MJfLRg7w@mail.gmail.com>
<CAM9w-_=c=82+5ESAEMXUQf5NOrfouZ5JxJMfr+56vJAfGz2DrA@mail.gmail.com>
<CA+OCxoxAhWEF5tQQy84QoOYB=ed--o--ppBoXZ1dwvWNBoNMQw@mail.gmail.com>
<CAM9w-_nUfackUWYNZDRcJ43D6ftYWnWkRsxvcyZVDzABkqza8A@mail.gmail.com>
<CA+OCxoyRgrQ=83Wo1YwsR9oSHE87=-20uP7d7zmF9F685CW2sA@mail.gmail.com>
<CAM9w-_ngHm2XpfXeA+RtWhifsHmN06rOEYwg5UrZBc=ZOum1Ug@mail.gmail.com>
<CA+OCxoyn32gZu78cc-p7Cr3zdsyviBVYD+X59W0-u+DTVUOzJg@mail.gmail.com>
<CAM9w-_=xKk35uQBYM+yZGkttryodU1CPF4n=jYEezKDm4GD_qw@mail.gmail.com>
<CA+OCxoxuS2GXK=LxREpniK6hHQ=d2G2FBpjD7JSohxZJpQwxPA@mail.gmail.com>
<CAM9w-_kPsWEodXtznzA0ZwXSO3d8t2agg5sUzF=2EEsGfakuMw@mail.gmail.com>
<CA+OCxoz7fFQn5ADzLbohXknMNk-1kfeTjAgCO4NrsXZPRiFVNw@mail.gmail.com>
<CAM9w-_mk04Rz6KVvQtmwiHs4s+OFDsDgCKebUO-vT0W2DfGi4w@mail.gmail.com>
<CA+OCxoyQtqqWhfNY5Um907AocAt1GsLcWvFJGjr5FLLeoWDCkA@mail.gmail.com>
<CAM9w-_n6LtJTXYhmrcYqvyvzXkshDMM3Z0yyKUWU4Vdmg7tNvw@mail.gmail.com>
<CAM9w-_=g0GJTyF8QegHJtKojcghcE+xzDPQUig_YzUnnH3nyWg@mail.gmail.com>
<CA+OCxox-DmkBptAXwiwi_GMCsow0dXzgE2C-wOyMZqGmUd3vCQ@mail.gmail.com>
<CAM9w-_niwZ+VD4awqOLp1D5eEFtE_G8wWciWC_sg8N2yvER=Nw@mail.gmail.com>
<CAM9w-_=8k4gBg_kji7KQ7FO=j18tCU5kDzu1YGGF3Xw7XEfK7A@mail.gmail.com>
<CA+OCxowxJghshFjXCtOUiRa0qqS93pC4m-bPRhdb7Q3AWe+HnQ@mail.gmail.com>
Hi Dave,
Attached is the updated patch. (Playing with encodings is not at all fun :(
)
On Tue, Jun 19, 2018 at 2:23 AM, Dave Page <[email protected]> wrote:
> Hi
>
> On Mon, Jun 18, 2018 at 2:14 PM, Aditya Toshniwal <aditya.toshniwal@
> enterprisedb.com> wrote:
>
>> Hi Hackers,
>>
>> Attached is the updated patch which includes the fix for Download CSV
>> fail in SQL_ASCII database, which is RM3250
>> <https://redmine.postgresql.org/issues/3250;
>> This should fix RM3289 and RM3250. As they interrelated, sending the
>> combined patch.
>> Kindly review.
>>
>
> This is definitely looking better - both view and save now work as
> expected. However, using the test data the I posted upthread, if I try to
> edit a value (in this case by adding a couple of chars to the end of the
> data in row 2) I get:
>
It should fix the error.
>
> 2018-06-18 16:41:40,895: SQL pgadmin: Execute (void) for server #1 -
> DB:ascii (Query-id: 3093186):
> UPDATE public.ascii SET
> data = %(data)s::text WHERE
> id = '2';
> 2018-06-18 16:41:41,027: INFO werkzeug: 127.0.0.1 - - [18/Jun/2018
> 16:41:41] "POST /sqleditor/save/2805058 HTTP/1.1" 500 -
> 2018-06-18 16:41:41,042: ERROR werkzeug: Error on request:
> Traceback (most recent call last):
> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/werkzeug/serving.py",
> line 270, in run_wsgi
> execute(self.server.app)
> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/werkzeug/serving.py",
> line 258, in execute
> application_iter = app(environ, start_response)
> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
> line 1997, in __call__
> return self.wsgi_app(environ, start_response)
> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
> line 1985, in wsgi_app
> response = self.handle_exception(e)
> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
> line 1540, in handle_exception
> reraise(exc_type, exc_value, tb)
> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
> line 1982, in wsgi_app
> response = self.full_dispatch_request()
> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
> line 1614, in full_dispatch_request
> rv = self.handle_user_exception(e)
> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
> line 1517, in handle_user_exception
> reraise(exc_type, exc_value, tb)
> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
> line 1612, in full_dispatch_request
> rv = self.dispatch_request()
> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
> line 1598, in dispatch_request
> return self.view_functions[rule.endpoint](**req.view_args)
> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask_login.py",
> line 792, in decorated_view
> return func(*args, **kwargs)
> File "/Users/dpage/git/pgadmin4/web/pgadmin/tools/sqleditor/__init__.py",
> line 776, in save
> default_conn)
> File "/Users/dpage/git/pgadmin4/web/pgadmin/tools/sqleditor/command.py",
> line 829, in save
> item['sql'], item['data'])
> File "/Users/dpage/git/pgadmin4/web/pgadmin/utils/driver/psycopg2/connection.py",
> line 975, in execute_void
> self.__internal_blocking_execute(cur, query, params)
> File "/Users/dpage/git/pgadmin4/web/pgadmin/utils/driver/psycopg2/connection.py",
> line 629, in __internal_blocking_execute
> cur.execute(query, params)
> File "/Users/dpage/git/pgadmin4/web/pgadmin/utils/driver/psycopg2/cursor.py",
> line 176, in execute
> return _cursor.execute(self, query, params)
> UnicodeEncodeError: 'ascii' codec can't encode characters in position
> 19-21: ordinal not in range(128)
>
>
>>
>> On Fri, Jun 15, 2018 at 2:33 PM, Aditya Toshniwal <
>> [email protected]> wrote:
>>
>>> Hi Dave,
>>>
>>> On Fri, Jun 15, 2018 at 2:08 PM, Dave Page <[email protected]> wrote:
>>>
>>>> Hi
>>>>
>>>> On Thu, Jun 14, 2018 at 7:05 AM, Aditya Toshniwal <
>>>> [email protected]> wrote:
>>>>
>>>>> I am sorry I missed the attachment. :(
>>>>> PFA.
>>>>>
>>>>
>>>> It looks like the encoding names are getting munged somewhere. I see
>>>> you've accounted for that to some degree in connection.py (you have both
>>>> SQL_ASCII/MULE_INTERNAL and SQLASCII/MULEINTERNAL), however it doesn't seem
>>>> to be enough as I'm getting the following error when trying to download CSV
>>>> from the query tool. Can we ensure that conn.encoding contains an un-munged
>>>> value at all times, or is that coming from psycopg2?
>>>>
>>> That is done by pyscopg2 and conn.encoding is a psycopg2 connection
>>> property.
>>>
>>>
>>>>
>>>> 2018-06-15 09:32:28,799: INFO werkzeug: 127.0.0.1 - - [15/Jun/2018
>>>> 09:32:28] "GET /sqleditor/query_tool/download
>>>> /2732923?query=SELECT%20*%20FROM%20public.sql_ascii%0AORDER%
>>>> 20BY%20id%20ASC%20&filename=sql_ascii.csv HTTP/1.1" 500 -
>>>> 2018-06-15 09:32:28,801: ERROR werkzeug: Error on request:
>>>> Traceback (most recent call last):
>>>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/werkzeug/serving.py",
>>>> line 270, in run_wsgi
>>>> execute(self.server.app)
>>>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/werkzeug/serving.py",
>>>> line 260, in execute
>>>> for data in application_iter:
>>>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/werkzeug/wsgi.py",
>>>> line 870, in __next__
>>>> return self._next()
>>>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/werkzeug/wrappers.py",
>>>> line 82, in _iter_encoded
>>>> for item in iterable:
>>>> File "/Users/dpage/git/pgadmin4/web/pgadmin/utils/driver/psycopg2/connection.py",
>>>> line 752, in gen
>>>> column_name = column_name.decode(conn_encoding)
>>>> LookupError: unknown encoding: SQLASCII
>>>>
>>>
>>> This is because there is code bug here. Below is code used to decode a
>>> column name. Connection encoding and python encoding are two different
>>> things. Python does not know what SQLASCII is. This will work with UTF-8
>>> because python has decoder with same name. I tried to download CSV with the
>>> original code without changes and it fails there too. I will fix this and
>>> will send the updated patch. I should have checked this.
>>> conn_encoding = cur.connection.encoding
>>> column_name = column_name.decode(conn_encoding)
>>>
>>>
>>>>
>>>> --
>>>> Dave Page
>>>> Blog: http://pgsnake.blogspot.com
>>>> Twitter: @pgsnake
>>>>
>>>> EnterpriseDB UK: http://www.enterprisedb.com
>>>> The Enterprise PostgreSQL Company
>>>>
>>>
>>>
>>>
>>> --
>>> Thanks and Regards,
>>> Aditya Toshniwal
>>> Software Engineer | EnterpriseDB Software Solutions | Pune
>>> "Don't Complain about Heat, Plant a tree"
>>>
>>
>>
>>
>> --
>> Thanks and Regards,
>> Aditya Toshniwal
>> Software Engineer | EnterpriseDB Software Solutions | Pune
>> "Don't Complain about Heat, Plant a tree"
>>
>
>
>
> --
> Dave Page
> Blog: http://pgsnake.blogspot.com
> Twitter: @pgsnake
>
> EnterpriseDB UK: http://www.enterprisedb.com
> The Enterprise PostgreSQL Company
>
--
Thanks and Regards,
Aditya Toshniwal
Software Engineer | EnterpriseDB Software Solutions | Pune
"Don't Complain about Heat, Plant a tree"
Attachments:
[application/octet-stream] RM3289_3250.patch (14.3K, 3-RM3289_3250.patch)
download | inline diff:
diff --git a/web/pgadmin/tools/sqleditor/__init__.py b/web/pgadmin/tools/sqleditor/__init__.py
index a9460dd7..d8f2ce67 100644
--- a/web/pgadmin/tools/sqleditor/__init__.py
+++ b/web/pgadmin/tools/sqleditor/__init__.py
@@ -574,7 +574,8 @@ def poll(trans_id):
'client_primary_key': client_primary_key,
'has_oids': has_oids,
'oids': oids
- }
+ },
+ encoding=conn.python_encoding
)
@@ -646,7 +647,8 @@ def fetch(trans_id, fetch_all=None):
'has_more_rows': has_more_rows,
'rows_fetched_from': rows_fetched_from,
'rows_fetched_to': rows_fetched_to
- }
+ },
+ encoding=conn.python_encoding
)
diff --git a/web/pgadmin/tools/sqleditor/tests/test_encoding_charset.py b/web/pgadmin/tools/sqleditor/tests/test_encoding_charset.py
new file mode 100644
index 00000000..8a8d45e3
--- /dev/null
+++ b/web/pgadmin/tools/sqleditor/tests/test_encoding_charset.py
@@ -0,0 +1,113 @@
+##########################################################################
+#
+# pgAdmin 4 - PostgreSQL Tools
+#
+# Copyright (C) 2013 - 2018, The pgAdmin Development Team
+# This software is released under the PostgreSQL Licence
+#
+##########################################################################
+
+from pgadmin.utils.route import BaseTestGenerator
+from pgadmin.browser.server_groups.servers.databases.tests import utils as \
+ database_utils
+from regression import parent_node_dict
+from regression.python_test_utils import test_utils
+import json
+from pgadmin.utils import server_utils
+
+
+class TestEncodingCharset(BaseTestGenerator):
+ """
+ This class validates character support in pgAdmin4 for
+ different PostgresDB encodings
+ """
+ skip_on_database = ['gpdb']
+ scenarios = [
+ (
+ 'With Encoding UTF8',
+ dict(
+ db_encoding='UTF8',
+ lc_collate='C',
+ test_str='A'
+ )),
+ (
+ 'With Encoding WIN1252',
+ dict(
+ db_encoding='WIN1252',
+ lc_collate='C',
+ test_str='A'
+ )),
+ (
+ 'With Encoding EUC_CN',
+ dict(
+ db_encoding='EUC_CN',
+ lc_collate='C',
+ test_str='A'
+ )),
+ (
+ 'With Encoding SQL_ASCII',
+ dict(
+ db_encoding='SQL_ASCII',
+ lc_collate='C',
+ test_str='\\255'
+ )),
+ ]
+
+ def setUp(self):
+ self.encode_db_name = 'encoding_' + self.db_encoding
+ self.encode_sid = self.server_information['server_id']
+
+ server_con = server_utils.connect_server(self, self.encode_sid)
+ if hasattr(self, 'skip_on_database'):
+ if 'data' in server_con and 'type' in server_con['data']:
+ if server_con['data']['type'] in self.skip_on_database:
+ self.skipTest('cannot run in: %s' %
+ server_con['data']['type'])
+
+ self.encode_did = test_utils.create_database(
+ self.server, self.encode_db_name,
+ (self.db_encoding, self.lc_collate))
+
+ def runTest(self):
+
+ db_con = database_utils.connect_database(self,
+ test_utils.SERVER_GROUP,
+ self.encode_sid,
+ self.encode_did)
+ if not db_con["info"] == "Database connected.":
+ raise Exception("Could not connect to the database.")
+
+ # Initialize query tool
+ url = '/datagrid/initialize/query_tool/{0}/{1}/{2}'.format(
+ test_utils.SERVER_GROUP, self.encode_sid, self.encode_did)
+ response = self.tester.post(url)
+ self.assertEquals(response.status_code, 200)
+
+ response_data = json.loads(response.data.decode('utf-8'))
+ self.trans_id = response_data['data']['gridTransId']
+
+ # Check character
+ url = "/sqleditor/query_tool/start/{0}".format(self.trans_id)
+ sql = "select E'{0}';".format(self.test_str)
+ response = self.tester.post(url, data=json.dumps({"sql": sql}),
+ content_type='html/json')
+ self.assertEquals(response.status_code, 200)
+ url = '/sqleditor/poll/{0}'.format(self.trans_id)
+ response = self.tester.get(url)
+ self.assertEquals(response.status_code, 200)
+ response_data = json.loads(response.data.decode('utf-8'))
+ self.assertEquals(response_data['data']['rows_fetched_to'], 1)
+
+ database_utils.disconnect_database(self, self.encode_sid,
+ self.encode_did)
+
+ def tearDown(self):
+ main_conn = test_utils.get_db_connection(
+ self.server['db'],
+ self.server['username'],
+ self.server['db_password'],
+ self.server['host'],
+ self.server['port'],
+ self.server['sslmode']
+ )
+ test_utils.drop_database(main_conn, self.encode_db_name)
diff --git a/web/pgadmin/utils/ajax.py b/web/pgadmin/utils/ajax.py
index 9b44aa3e..c19f778a 100644
--- a/web/pgadmin/utils/ajax.py
+++ b/web/pgadmin/utils/ajax.py
@@ -45,7 +45,8 @@ def get_no_cache_header():
def make_json_response(
- success=1, errormsg='', info='', result=None, data=None, status=200
+ success=1, errormsg='', info='', result=None, data=None, status=200,
+ encoding='utf-8'
):
"""Create a HTML response document describing the results of a request and
containing the data."""
@@ -58,7 +59,7 @@ def make_json_response(
return Response(
response=json.dumps(doc, cls=DataTypeJSONEncoder,
- separators=(',', ':')),
+ separators=(',', ':'), encoding=encoding),
status=status,
mimetype="application/json",
headers=get_no_cache_header()
diff --git a/web/pgadmin/utils/driver/psycopg2/connection.py b/web/pgadmin/utils/driver/psycopg2/connection.py
index cfd161a0..6f522da6 100644
--- a/web/pgadmin/utils/driver/psycopg2/connection.py
+++ b/web/pgadmin/utils/driver/psycopg2/connection.py
@@ -50,7 +50,6 @@ else:
_ = gettext
-
# Register global type caster which will be applicable to all connections.
register_global_typecasters()
@@ -397,10 +396,29 @@ class Connection(BaseConnection):
if self.use_binary_placeholder:
register_binary_typecasters(self.conn)
- status = _execute(cur, "SET DateStyle=ISO;"
- "SET client_min_messages=notice;"
- "SET bytea_output=escape;"
- "SET client_encoding='UNICODE';")
+ if self.conn.encoding in ('SQL_ASCII', 'SQLASCII',
+ 'MULE_INTERNAL', 'MULEINTERNAL'):
+ status = _execute(cur, "SET DateStyle=ISO;"
+ "SET client_min_messages=notice;"
+ "SET bytea_output=escape;"
+ "SET client_encoding='{0}';"
+ .format(self.conn.encoding))
+ self.python_encoding = 'raw_unicode_escape'
+ else:
+ status = _execute(cur, "SET DateStyle=ISO;"
+ "SET client_min_messages=notice;"
+ "SET bytea_output=escape;"
+ "SET client_encoding='UNICODE';")
+ self.python_encoding = 'utf-8'
+
+ # Replace the python encoding for original name and renamed encodings
+ # psycopg2 removes the underscore in conn.encoding
+ # Setting the encodings dict value will only help for select statements
+ # because for parameterized DML, param values are converted based on
+ # python encoding of pyscopg2s internal encodings dict.
+ for key, val in encodings.items():
+ if key.replace('_', '') == self.conn.encoding:
+ encodings[key] = self.python_encoding
if status is not None:
self.conn.close()
@@ -598,6 +616,21 @@ WHERE
return True, cur
+ def escape_params_sqlascii(self, params):
+ # The data is unescaped using string_typecasters when selected
+ # We need to esacpe the data so that it does not fail when
+ # it is encoded with python ascii
+ # unicode_escape helps in escaping and unescaping
+ if self.conn.encoding in ('SQL_ASCII', 'SQLASCII',
+ 'MULE_INTERNAL', 'MULEINTERNAL')\
+ and params is not None and type(params) == dict:
+ params = {
+ key: val.encode('unicode_escape')
+ .decode('raw_unicode_escape')
+ for key, val in params.items()
+ }
+ return params
+
def __internal_blocking_execute(self, cur, query, params):
"""
This function executes the query using cursor's execute function,
@@ -617,6 +650,7 @@ WHERE
else:
query = query.encode('utf-8')
+ params = self.escape_params_sqlascii(params)
cur.execute(query, params)
if self.async == 1:
self._wait(cur.connection)
@@ -734,7 +768,7 @@ WHERE
header = []
json_columns = []
- conn_encoding = cur.connection.encoding
+ conn_encoding = encodings[cur.connection.encoding]
for c in cur.ordered_description():
# This is to handle the case in which column name is non-ascii
@@ -880,6 +914,9 @@ WHERE
else:
query = query.encode('utf-8')
+ # Convert the params based on python_encoding
+ params = self.escape_params_sqlascii(params)
+
self.__async_cursor = None
status, cur = self.__cursor()
diff --git a/web/pgadmin/utils/driver/psycopg2/typecast.py b/web/pgadmin/utils/driver/psycopg2/typecast.py
index f1366049..de2e6325 100644
--- a/web/pgadmin/utils/driver/psycopg2/typecast.py
+++ b/web/pgadmin/utils/driver/psycopg2/typecast.py
@@ -164,46 +164,37 @@ def register_global_typecasters():
def register_string_typecasters(connection):
- if connection.encoding != 'UTF8':
- # In python3 when database encoding is other than utf-8 and client
- # encoding is set to UNICODE then we need to map data from database
- # encoding to utf-8.
- # This is required because when client encoding is set to UNICODE then
- # psycopg assumes database encoding utf-8 and not the actual encoding.
- # Not sure whether it's bug or feature in psycopg for python3.
+ # raw_unicode_escape used for SQL ASCII will escape the
+ # characters. Here we unescape them using unicode_escape
+ # and send ahead. When insert update is done, the characters
+ # are escaped again and sent to the DB.
+ if connection.encoding in ('SQL_ASCII', 'SQLASCII',
+ 'MULE_INTERNAL', 'MULEINTERNAL'):
if sys.version_info >= (3,):
- def return_as_unicode(value, cursor):
+ def non_ascii_escape(value, cursor):
if value is None:
return None
- # Treat value as byte sequence of database encoding and then
- # decode it as utf-8 to get correct unicode value.
return bytes(
value, encodings[cursor.connection.encoding]
- ).decode('utf-8')
-
- unicode_type = psycopg2.extensions.new_type(
- # "char", name, text, character, character varying
- (19, 18, 25, 1042, 1043, 0),
- 'UNICODE', return_as_unicode)
+ ).decode('unicode_escape')
else:
- def return_as_unicode(value, cursor):
+ def non_ascii_escape(value, cursor):
if value is None:
return None
- # Decode it as utf-8 to get correct unicode value.
- return value.decode('utf-8')
+ return value.decode('unicode_escape')
- unicode_type = psycopg2.extensions.new_type(
- # "char", name, text, character, character varying
- (19, 18, 25, 1042, 1043, 0),
- 'UNICODE', return_as_unicode)
+ unicode_type = psycopg2.extensions.new_type(
+ # "char", name, text, character, character varying
+ (19, 18, 25, 1042, 1043, 0),
+ 'UNICODE', non_ascii_escape)
unicode_array_type = psycopg2.extensions.new_array_type(
# "char"[], name[], text[], character[], character varying[]
(1002, 1003, 1009, 1014, 1015, 0
), 'UNICODEARRAY', unicode_type)
- psycopg2.extensions.register_type(unicode_type)
- psycopg2.extensions.register_type(unicode_array_type)
+ psycopg2.extensions.register_type(unicode_type, connection)
+ psycopg2.extensions.register_type(unicode_array_type, connection)
def register_binary_typecasters(connection):
diff --git a/web/regression/python_test_utils/test_utils.py b/web/regression/python_test_utils/test_utils.py
index 3e517b61..464a09e1 100644
--- a/web/regression/python_test_utils/test_utils.py
+++ b/web/regression/python_test_utils/test_utils.py
@@ -116,7 +116,7 @@ def clear_node_info_dict():
del node_info_dict[node][:]
-def create_database(server, db_name):
+def create_database(server, db_name, encoding=None):
"""This function used to create database and returns the database id"""
try:
connection = get_db_connection(
@@ -130,8 +130,14 @@ def create_database(server, db_name):
old_isolation_level = connection.isolation_level
connection.set_isolation_level(0)
pg_cursor = connection.cursor()
- pg_cursor.execute(
- '''CREATE DATABASE "%s" TEMPLATE template0''' % db_name)
+ if encoding is None:
+ pg_cursor.execute(
+ '''CREATE DATABASE "%s" TEMPLATE template0''' % db_name)
+ else:
+ pg_cursor.execute(
+ '''CREATE DATABASE "%s" TEMPLATE template0
+ ENCODING='%s' LC_COLLATE='%s' LC_CTYPE='%s' ''' %
+ (db_name, encoding[0], encoding[1], encoding[1]))
connection.set_isolation_level(old_isolation_level)
connection.commit()
view thread (38+ messages) latest in thread
reply
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: [email protected]
Cc: [email protected], [email protected], [email protected]
Subject: Re: [pgAdmin4][RM#3289] Can't query SQL_ASCII database.
In-Reply-To: <CAM9w-_k7FeHMwcOd1RqkCC64pF2b3=WHXpM14YJZ0b45tWUdBw@mail.gmail.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox