public inbox for [email protected]
help / color / mirror / Atom feedFrom: Aditya Toshniwal <[email protected]>
To: pgadmin-hackers <[email protected]>
Subject: Re: [pgAdmin4][RM#3289] Can't query SQL_ASCII database.
Date: Wed, 30 May 2018 12:36:32 +0530
Message-ID: <CAM9w-_mXdJhuRFn87yAuqWtfrhnRgw1KvU2PpthL-hbnOt2QpA@mail.gmail.com> (raw)
In-Reply-To: <CAM9w-_mTe0gW+2cdhMutMWyBsRt7Om3G_Em126Ryntf0nOiHbA@mail.gmail.com>
References: <CAM9w-_ntCUguZp2rVJXPsDvWJbWneAz4-AHJoYB+nUTSYMXc9g@mail.gmail.com>
<CA+OCxozRjJY6RWrXpMLD6-dWnm2LdPs6bS+VmJa403vxVoLwhg@mail.gmail.com>
<CAM9w-_m-cJax16zqBonx-d0-HSy5dNoOzB13HWAad5Xv0Wu=CA@mail.gmail.com>
<CANxYE3+qrGeZYzdYEtzxS8i2P+325V5E+n1xVpyzTwn7vRqRDg@mail.gmail.com>
<CAM9w-_mTe0gW+2cdhMutMWyBsRt7Om3G_Em126Ryntf0nOiHbA@mail.gmail.com>
Hi Hackers,
PFA updated patch after all the permutations, combinations for encoding for
SQL_ASCII database. Also fixed a small glitch for sql editor connection
status check.
Please note, ERROR: invalid byte sequence for encoding "UTF8": 0xe9 0x73 is
a Postgres DB error and not pgAdmin4 error.
You need to change client_encoding to the appropriate. After changing
client_encoding using command - set client_encoding='XYZ', it will give not
give error.
Thanks and Regards,
Aditya Toshniwal
Software Engineer | EnterpriseDB Software Solutions | Pune
"Don't Complain about Heat, Plant a tree"
On Wed, May 23, 2018 at 10:13 AM, Aditya Toshniwal <
[email protected]> wrote:
> Thank you Victoria, Anthony.
>
> Thanks and Regards,
> Aditya Toshniwal
> Software Engineer | EnterpriseDB Software Solutions | Pune
> "Don't Complain about Heat, Plant a tree"
>
> On Tue, May 22, 2018 at 7:15 PM, Victoria Henry <[email protected]> wrote:
>
>> Hi Aditya,
>>
>> We made a minor change to make the patch so the python linter can pass.
>> Attached is the change we made.
>> Everything else looks good.
>>
>> Sincerely,
>>
>> Victoria & Anthony
>>
>> On Tue, May 22, 2018 at 4:46 AM Aditya Toshniwal <
>> [email protected]> wrote:
>>
>>> Hi,
>>>
>>> PFA updated patch. Linter issues are fixed ( we dont have any linter
>>> setup for python :-( )
>>> Regarding test cases, they run successfully on my system and the reason
>>> it failed for pivotal is timeout exception. I am sorry I can't help with
>>> that.
>>>
>>> Traceback (most recent call last):
>>> File "/tmp/build/a453582b/pgadmin-repo/web/pgadmin/feature_tests/keyboard_shortcut_test.py",
>>> line 52, in runTest
>>> self._check_shortcuts()
>>> File "/tmp/build/a453582b/pgadmin-repo/web/pgadmin/feature_tests/keyboard_shortcut_test.py",
>>> line 77, in _check_shortcuts
>>> ") and contains(@class, 'open')]")
>>> File "/root/.pyenv/versions/pgadmin36/lib/python3.6/site-packages
>>> /selenium/webdriver/support/wait.py", line 80, in until
>>> raise TimeoutException(message, screen, stacktrace)
>>> selenium.common.exceptions.TimeoutException: Message:
>>>
>>> Thanks and Regards,
>>> Aditya Toshniwal
>>> Software Engineer | EnterpriseDB Software Solutions | Pune
>>> "Don't Complain about Heat, Plant a tree"
>>>
>>> On Tue, May 22, 2018 at 1:37 PM, Dave Page <[email protected]> wrote:
>>>
>>>> Hi
>>>>
>>>> Pivotal's buildbot is showing problems with this patch:
>>>>
>>>> https://gpdb-dev.bosh.pivotalci.info/teams/pgadmin/pipelines
>>>> /pgadmin-patch/jobs/run-linter/builds/66 (linter failed)
>>>> https://gpdb-dev.bosh.pivotalci.info/teams/pgadmin/pipelines
>>>> /pgadmin-patch/jobs/run-tests/builds/84 (tests failed)
>>>>
>>>>
>>>> On Tue, May 22, 2018 at 7:05 AM, Aditya Toshniwal <
>>>> [email protected]> wrote:
>>>>
>>>>> Hi Hackers,
>>>>>
>>>>> PFA patch for RM#3289 where decode error was thrown on querying a
>>>>> SQL_ASCII database table. Please note, this problem occurs only on windows.
>>>>> Sample insert - insert into test_tab values ('é');
>>>>>
>>>>> psycopg2 has a encodings dictionary where Postgres Database Encodings
>>>>> are mapped to python equivalent. It uses 'ascii' decoder of python to
>>>>> decode for SQL_ASCII encoding. If data has characters beyond the limit of
>>>>> ascii then it failed. The solution would be to use utf_8 decoder instead of
>>>>> ascii. I tried setting the client_encoding using
>>>>> set_client_encoding('UTF8') method of a psycopg2 connection but no luck
>>>>> (also its not allowed for async connection). I also tried executing "SET
>>>>> CLIENT_ENCODING='UTF8'" but it didn't work too.
>>>>> So, as in the patch, I had to set encodings dict value directly to
>>>>> 'utf_8' and it seems to be working. Please note, the same is added to
>>>>> psycopg3 milestones
>>>>> https://github.com/psycopg/psycopg2/milestone/4
>>>>>
>>>>> Also fixed a small glitch for sql editor connection status check.
>>>>>
>>>>> Kindly review.
>>>>>
>>>>> Thanks and Regards,
>>>>> Aditya Toshniwal
>>>>> Software Engineer | EnterpriseDB Software Solutions | Pune
>>>>> "Don't Complain about Heat, Plant a tree"
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Dave Page
>>>> Blog: http://pgsnake.blogspot.com
>>>> Twitter: @pgsnake
>>>>
>>>> EnterpriseDB UK: http://www.enterprisedb.com
>>>> The Enterprise PostgreSQL Company
>>>>
>>>
>>>
>
Attachments:
[image/png] image.png (8.7K, 3-image.png)
download | view image
[image/png] image.png (9.0K, 4-image.png)
download | view image
[application/octet-stream] RM3289.patch (2.4K, 5-RM3289.patch)
download | inline diff:
diff --git a/web/pgadmin/tools/sqleditor/__init__.py b/web/pgadmin/tools/sqleditor/__init__.py
index c72505a4..ead100e7 100644
--- a/web/pgadmin/tools/sqleditor/__init__.py
+++ b/web/pgadmin/tools/sqleditor/__init__.py
@@ -1476,14 +1476,19 @@ def query_tool_status(trans_id):
if conn and trans_obj and session_obj:
status = conn.transaction_status()
- return make_json_response(
- data={
- 'status': status,
- 'message': gettext(
- CONNECTION_STATUS_MESSAGE_MAPPING.get(status)
- )
- }
- )
+ if status is not None:
+ return make_json_response(
+ data={
+ 'status': status,
+ 'message': gettext(
+ CONNECTION_STATUS_MESSAGE_MAPPING.get(status)
+ )
+ }
+ )
+ else:
+ return internal_server_error(
+ errormsg=gettext("Transaction status check failed.")
+ )
else:
return internal_server_error(
errormsg=gettext("Transaction status check failed.")
diff --git a/web/pgadmin/utils/driver/psycopg2/connection.py b/web/pgadmin/utils/driver/psycopg2/connection.py
index 315631c0..ac85f9dc 100644
--- a/web/pgadmin/utils/driver/psycopg2/connection.py
+++ b/web/pgadmin/utils/driver/psycopg2/connection.py
@@ -50,6 +50,12 @@ else:
_ = gettext
+# Replace default ascii encoder with unicode-escape
+# which translates characters to unicode format.
+# Escape special characters to ASCII based on unicode
+encodings['SQL_ASCII'] = 'unicode-escape'
+encodings['SQLASCII'] = 'unicode-escape'
+
# Register global type caster which will be applicable to all connections.
register_global_typecasters()
diff --git a/web/pgadmin/utils/driver/psycopg2/typecast.py b/web/pgadmin/utils/driver/psycopg2/typecast.py
index f1366049..ccebb2f9 100644
--- a/web/pgadmin/utils/driver/psycopg2/typecast.py
+++ b/web/pgadmin/utils/driver/psycopg2/typecast.py
@@ -164,7 +164,7 @@ def register_global_typecasters():
def register_string_typecasters(connection):
- if connection.encoding != 'UTF8':
+ if connection.encoding not in ('UTF8', 'SQLASCII', 'SQL_ASCII'):
# In python3 when database encoding is other than utf-8 and client
# encoding is set to UNICODE then we need to map data from database
# encoding to utf-8.
view thread (38+ messages) latest in thread
reply
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: [email protected]
Cc: [email protected]
Subject: Re: [pgAdmin4][RM#3289] Can't query SQL_ASCII database.
In-Reply-To: <CAM9w-_mXdJhuRFn87yAuqWtfrhnRgw1KvU2PpthL-hbnOt2QpA@mail.gmail.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox