public inbox for [email protected]
help / color / mirror / Atom feed[pgAdmin4][RM#3289] Can't query SQL_ASCII database.
38+ messages / 4 participants
[nested] [flat]
* [pgAdmin4][RM#3289] Can't query SQL_ASCII database.
@ 2018-05-22 06:05 Aditya Toshniwal <[email protected]>
0 siblings, 1 reply; 38+ messages in thread
From: Aditya Toshniwal @ 2018-05-22 06:05 UTC (permalink / raw)
To: pgadmin-hackers
Hi Hackers,
PFA patch for RM#3289 where decode error was thrown on querying a SQL_ASCII
database table. Please note, this problem occurs only on windows.
Sample insert - insert into test_tab values ('é');
psycopg2 has a encodings dictionary where Postgres Database Encodings are
mapped to python equivalent. It uses 'ascii' decoder of python to decode
for SQL_ASCII encoding. If data has characters beyond the limit of ascii
then it failed. The solution would be to use utf_8 decoder instead of
ascii. I tried setting the client_encoding using
set_client_encoding('UTF8') method of a psycopg2 connection but no luck
(also its not allowed for async connection). I also tried executing "SET
CLIENT_ENCODING='UTF8'" but it didn't work too.
So, as in the patch, I had to set encodings dict value directly to 'utf_8'
and it seems to be working. Please note, the same is added to psycopg3
milestones
https://github.com/psycopg/psycopg2/milestone/4
Also fixed a small glitch for sql editor connection status check.
Kindly review.
Thanks and Regards,
Aditya Toshniwal
Software Engineer | EnterpriseDB Software Solutions | Pune
"Don't Complain about Heat, Plant a tree"
Attachments:
[application/octet-stream] RM3289.patch (1.1K, 3-RM3289.patch)
download | inline diff:
diff --git a/web/pgadmin/tools/sqleditor/__init__.py b/web/pgadmin/tools/sqleditor/__init__.py
index c72505a4..aec3faba 100644
--- a/web/pgadmin/tools/sqleditor/__init__.py
+++ b/web/pgadmin/tools/sqleditor/__init__.py
@@ -1474,7 +1474,7 @@ def query_tool_status(trans_id):
errormsg=error_msg
)
- if conn and trans_obj and session_obj:
+ if conn and trans_obj and session_obj and conn.transaction_status() is not None:
status = conn.transaction_status()
return make_json_response(
data={
diff --git a/web/pgadmin/utils/driver/psycopg2/connection.py b/web/pgadmin/utils/driver/psycopg2/connection.py
index 315631c0..ea90e62f 100644
--- a/web/pgadmin/utils/driver/psycopg2/connection.py
+++ b/web/pgadmin/utils/driver/psycopg2/connection.py
@@ -49,6 +49,9 @@ else:
_ = gettext
+# Use utf_8 decoder instead of ascii for SQL_ASCII database to avoid decode errors
+encodings['SQL_ASCII'] = 'utf_8'
+encodings['SQLASCII'] = 'utf_8'
# Register global type caster which will be applicable to all connections.
register_global_typecasters()
^ permalink raw reply [nested|flat] 38+ messages in thread
* Re: [pgAdmin4][RM#3289] Can't query SQL_ASCII database.
@ 2018-05-22 08:07 Dave Page <[email protected]>
parent: Aditya Toshniwal <[email protected]>
0 siblings, 1 reply; 38+ messages in thread
From: Dave Page @ 2018-05-22 08:07 UTC (permalink / raw)
To: Aditya Toshniwal <[email protected]>; +Cc: pgadmin-hackers
Hi
Pivotal's buildbot is showing problems with this patch:
https://gpdb-dev.bosh.pivotalci.info/teams/pgadmin/pipelines/pgadmin-patch/jobs/run-linter/builds/66
(linter failed)
https://gpdb-dev.bosh.pivotalci.info/teams/pgadmin/pipelines/pgadmin-patch/jobs/run-tests/builds/84
(tests failed)
On Tue, May 22, 2018 at 7:05 AM, Aditya Toshniwal <
[email protected]> wrote:
> Hi Hackers,
>
> PFA patch for RM#3289 where decode error was thrown on querying a
> SQL_ASCII database table. Please note, this problem occurs only on windows.
> Sample insert - insert into test_tab values ('é');
>
> psycopg2 has a encodings dictionary where Postgres Database Encodings are
> mapped to python equivalent. It uses 'ascii' decoder of python to decode
> for SQL_ASCII encoding. If data has characters beyond the limit of ascii
> then it failed. The solution would be to use utf_8 decoder instead of
> ascii. I tried setting the client_encoding using
> set_client_encoding('UTF8') method of a psycopg2 connection but no luck
> (also its not allowed for async connection). I also tried executing "SET
> CLIENT_ENCODING='UTF8'" but it didn't work too.
> So, as in the patch, I had to set encodings dict value directly to 'utf_8'
> and it seems to be working. Please note, the same is added to psycopg3
> milestones
> https://github.com/psycopg/psycopg2/milestone/4
>
> Also fixed a small glitch for sql editor connection status check.
>
> Kindly review.
>
> Thanks and Regards,
> Aditya Toshniwal
> Software Engineer | EnterpriseDB Software Solutions | Pune
> "Don't Complain about Heat, Plant a tree"
>
--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake
EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 38+ messages in thread
* Re: [pgAdmin4][RM#3289] Can't query SQL_ASCII database.
@ 2018-05-22 08:46 Aditya Toshniwal <[email protected]>
parent: Dave Page <[email protected]>
0 siblings, 1 reply; 38+ messages in thread
From: Aditya Toshniwal @ 2018-05-22 08:46 UTC (permalink / raw)
To: Dave Page <[email protected]>; +Cc: pgadmin-hackers
Hi,
PFA updated patch. Linter issues are fixed ( we dont have any linter setup
for python :-( )
Regarding test cases, they run successfully on my system and the reason it
failed for pivotal is timeout exception. I am sorry I can't help with that.
Traceback (most recent call last):
File
"/tmp/build/a453582b/pgadmin-repo/web/pgadmin/feature_tests/keyboard_shortcut_test.py",
line 52, in runTest
self._check_shortcuts()
File
"/tmp/build/a453582b/pgadmin-repo/web/pgadmin/feature_tests/keyboard_shortcut_test.py",
line 77, in _check_shortcuts
") and contains(@class, 'open')]")
File
"/root/.pyenv/versions/pgadmin36/lib/python3.6/site-packages/selenium/webdriver/support/wait.py",
line 80, in until
raise TimeoutException(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message:
Thanks and Regards,
Aditya Toshniwal
Software Engineer | EnterpriseDB Software Solutions | Pune
"Don't Complain about Heat, Plant a tree"
On Tue, May 22, 2018 at 1:37 PM, Dave Page <[email protected]> wrote:
> Hi
>
> Pivotal's buildbot is showing problems with this patch:
>
> https://gpdb-dev.bosh.pivotalci.info/teams/pgadmin/
> pipelines/pgadmin-patch/jobs/run-linter/builds/66 (linter failed)
> https://gpdb-dev.bosh.pivotalci.info/teams/pgadmin/
> pipelines/pgadmin-patch/jobs/run-tests/builds/84 (tests failed)
>
>
> On Tue, May 22, 2018 at 7:05 AM, Aditya Toshniwal <aditya.toshniwal@
> enterprisedb.com> wrote:
>
>> Hi Hackers,
>>
>> PFA patch for RM#3289 where decode error was thrown on querying a
>> SQL_ASCII database table. Please note, this problem occurs only on windows.
>> Sample insert - insert into test_tab values ('é');
>>
>> psycopg2 has a encodings dictionary where Postgres Database Encodings are
>> mapped to python equivalent. It uses 'ascii' decoder of python to decode
>> for SQL_ASCII encoding. If data has characters beyond the limit of ascii
>> then it failed. The solution would be to use utf_8 decoder instead of
>> ascii. I tried setting the client_encoding using
>> set_client_encoding('UTF8') method of a psycopg2 connection but no luck
>> (also its not allowed for async connection). I also tried executing "SET
>> CLIENT_ENCODING='UTF8'" but it didn't work too.
>> So, as in the patch, I had to set encodings dict value directly to
>> 'utf_8' and it seems to be working. Please note, the same is added to
>> psycopg3 milestones
>> https://github.com/psycopg/psycopg2/milestone/4
>>
>> Also fixed a small glitch for sql editor connection status check.
>>
>> Kindly review.
>>
>> Thanks and Regards,
>> Aditya Toshniwal
>> Software Engineer | EnterpriseDB Software Solutions | Pune
>> "Don't Complain about Heat, Plant a tree"
>>
>
>
>
> --
> Dave Page
> Blog: http://pgsnake.blogspot.com
> Twitter: @pgsnake
>
> EnterpriseDB UK: http://www.enterprisedb.com
> The Enterprise PostgreSQL Company
>
Attachments:
[text/x-patch] RM3289.patch (1.1K, 3-RM3289.patch)
download | inline diff:
diff --git a/web/pgadmin/tools/sqleditor/__init__.py b/web/pgadmin/tools/sqleditor/__init__.py
index c72505a..fa8b83f 100644
--- a/web/pgadmin/tools/sqleditor/__init__.py
+++ b/web/pgadmin/tools/sqleditor/__init__.py
@@ -1474,7 +1474,8 @@ def query_tool_status(trans_id):
errormsg=error_msg
)
- if conn and trans_obj and session_obj:
+ if conn and trans_obj and session_obj \
+ and conn.transaction_status() is not None:
status = conn.transaction_status()
return make_json_response(
data={
diff --git a/web/pgadmin/utils/driver/psycopg2/connection.py b/web/pgadmin/utils/driver/psycopg2/connection.py
index 315631c..8ca30b3 100644
--- a/web/pgadmin/utils/driver/psycopg2/connection.py
+++ b/web/pgadmin/utils/driver/psycopg2/connection.py
@@ -49,6 +49,10 @@ else:
_ = gettext
+# Use utf_8 decoder instead of psycopg2 default ascii for
+# SQL_ASCII database to avoid decode errors
+encodings['SQL_ASCII'] = 'utf_8'
+encodings['SQLASCII'] = 'utf_8'
# Register global type caster which will be applicable to all connections.
register_global_typecasters()
^ permalink raw reply [nested|flat] 38+ messages in thread
* Re: [pgAdmin4][RM#3289] Can't query SQL_ASCII database.
@ 2018-05-22 13:45 Victoria Henry <[email protected]>
parent: Aditya Toshniwal <[email protected]>
0 siblings, 1 reply; 38+ messages in thread
From: Victoria Henry @ 2018-05-22 13:45 UTC (permalink / raw)
To: Aditya Toshniwal <[email protected]>; +Cc: Dave Page <[email protected]>; pgadmin-hackers
Hi Aditya,
We made a minor change to make the patch so the python linter can pass.
Attached is the change we made.
Everything else looks good.
Sincerely,
Victoria & Anthony
On Tue, May 22, 2018 at 4:46 AM Aditya Toshniwal <
[email protected]> wrote:
> Hi,
>
> PFA updated patch. Linter issues are fixed ( we dont have any linter setup
> for python :-( )
> Regarding test cases, they run successfully on my system and the reason it
> failed for pivotal is timeout exception. I am sorry I can't help with that.
>
> Traceback (most recent call last):
> File
> "/tmp/build/a453582b/pgadmin-repo/web/pgadmin/feature_tests/keyboard_shortcut_test.py",
> line 52, in runTest
> self._check_shortcuts()
> File
> "/tmp/build/a453582b/pgadmin-repo/web/pgadmin/feature_tests/keyboard_shortcut_test.py",
> line 77, in _check_shortcuts
> ") and contains(@class, 'open')]")
> File
> "/root/.pyenv/versions/pgadmin36/lib/python3.6/site-packages/selenium/webdriver/support/wait.py",
> line 80, in until
> raise TimeoutException(message, screen, stacktrace)
> selenium.common.exceptions.TimeoutException: Message:
>
> Thanks and Regards,
> Aditya Toshniwal
> Software Engineer | EnterpriseDB Software Solutions | Pune
> "Don't Complain about Heat, Plant a tree"
>
> On Tue, May 22, 2018 at 1:37 PM, Dave Page <[email protected]> wrote:
>
>> Hi
>>
>> Pivotal's buildbot is showing problems with this patch:
>>
>>
>> https://gpdb-dev.bosh.pivotalci.info/teams/pgadmin/pipelines/pgadmin-patch/jobs/run-linter/builds/66
>> (linter failed)
>>
>> https://gpdb-dev.bosh.pivotalci.info/teams/pgadmin/pipelines/pgadmin-patch/jobs/run-tests/builds/84
>> (tests failed)
>>
>>
>> On Tue, May 22, 2018 at 7:05 AM, Aditya Toshniwal <
>> [email protected]> wrote:
>>
>>> Hi Hackers,
>>>
>>> PFA patch for RM#3289 where decode error was thrown on querying a
>>> SQL_ASCII database table. Please note, this problem occurs only on windows.
>>> Sample insert - insert into test_tab values ('é');
>>>
>>> psycopg2 has a encodings dictionary where Postgres Database Encodings
>>> are mapped to python equivalent. It uses 'ascii' decoder of python to
>>> decode for SQL_ASCII encoding. If data has characters beyond the limit of
>>> ascii then it failed. The solution would be to use utf_8 decoder instead of
>>> ascii. I tried setting the client_encoding using
>>> set_client_encoding('UTF8') method of a psycopg2 connection but no luck
>>> (also its not allowed for async connection). I also tried executing "SET
>>> CLIENT_ENCODING='UTF8'" but it didn't work too.
>>> So, as in the patch, I had to set encodings dict value directly to
>>> 'utf_8' and it seems to be working. Please note, the same is added to
>>> psycopg3 milestones
>>> https://github.com/psycopg/psycopg2/milestone/4
>>>
>>> Also fixed a small glitch for sql editor connection status check.
>>>
>>> Kindly review.
>>>
>>> Thanks and Regards,
>>> Aditya Toshniwal
>>> Software Engineer | EnterpriseDB Software Solutions | Pune
>>> "Don't Complain about Heat, Plant a tree"
>>>
>>
>>
>>
>> --
>> Dave Page
>> Blog: http://pgsnake.blogspot.com
>> Twitter: @pgsnake
>>
>> EnterpriseDB UK: http://www.enterprisedb.com
>> The Enterprise PostgreSQL Company
>>
>
>
Attachments:
[application/octet-stream] RM3289_v2.patch (1.1K, 3-RM3289_v2.patch)
download | inline diff:
diff --git a/web/pgadmin/tools/sqleditor/__init__.py b/web/pgadmin/tools/sqleditor/__init__.py
index c72505a4..9f41cf0a 100644
--- a/web/pgadmin/tools/sqleditor/__init__.py
+++ b/web/pgadmin/tools/sqleditor/__init__.py
@@ -1474,7 +1474,8 @@ def query_tool_status(trans_id):
errormsg=error_msg
)
- if conn and trans_obj and session_obj:
+ if conn and trans_obj and session_obj \
+ and conn.transaction_status() is not None:
status = conn.transaction_status()
return make_json_response(
data={
diff --git a/web/pgadmin/utils/driver/psycopg2/connection.py b/web/pgadmin/utils/driver/psycopg2/connection.py
index 315631c0..8ca30b39 100644
--- a/web/pgadmin/utils/driver/psycopg2/connection.py
+++ b/web/pgadmin/utils/driver/psycopg2/connection.py
@@ -49,6 +49,10 @@ else:
_ = gettext
+# Use utf_8 decoder instead of psycopg2 default ascii for
+# SQL_ASCII database to avoid decode errors
+encodings['SQL_ASCII'] = 'utf_8'
+encodings['SQLASCII'] = 'utf_8'
# Register global type caster which will be applicable to all connections.
register_global_typecasters()
^ permalink raw reply [nested|flat] 38+ messages in thread
* Re: [pgAdmin4][RM#3289] Can't query SQL_ASCII database.
@ 2018-05-23 04:43 Aditya Toshniwal <[email protected]>
parent: Victoria Henry <[email protected]>
0 siblings, 1 reply; 38+ messages in thread
From: Aditya Toshniwal @ 2018-05-23 04:43 UTC (permalink / raw)
To: Victoria Henry <[email protected]>; +Cc: Dave Page <[email protected]>; pgadmin-hackers
Thank you Victoria, Anthony.
Thanks and Regards,
Aditya Toshniwal
Software Engineer | EnterpriseDB Software Solutions | Pune
"Don't Complain about Heat, Plant a tree"
On Tue, May 22, 2018 at 7:15 PM, Victoria Henry <[email protected]> wrote:
> Hi Aditya,
>
> We made a minor change to make the patch so the python linter can pass.
> Attached is the change we made.
> Everything else looks good.
>
> Sincerely,
>
> Victoria & Anthony
>
> On Tue, May 22, 2018 at 4:46 AM Aditya Toshniwal <aditya.toshniwal@
> enterprisedb.com> wrote:
>
>> Hi,
>>
>> PFA updated patch. Linter issues are fixed ( we dont have any linter
>> setup for python :-( )
>> Regarding test cases, they run successfully on my system and the reason
>> it failed for pivotal is timeout exception. I am sorry I can't help with
>> that.
>>
>> Traceback (most recent call last):
>> File "/tmp/build/a453582b/pgadmin-repo/web/pgadmin/feature_
>> tests/keyboard_shortcut_test.py", line 52, in runTest
>> self._check_shortcuts()
>> File "/tmp/build/a453582b/pgadmin-repo/web/pgadmin/feature_
>> tests/keyboard_shortcut_test.py", line 77, in _check_shortcuts
>> ") and contains(@class, 'open')]")
>> File "/root/.pyenv/versions/pgadmin36/lib/python3.6/site-
>> packages/selenium/webdriver/support/wait.py", line 80, in until
>> raise TimeoutException(message, screen, stacktrace)
>> selenium.common.exceptions.TimeoutException: Message:
>>
>> Thanks and Regards,
>> Aditya Toshniwal
>> Software Engineer | EnterpriseDB Software Solutions | Pune
>> "Don't Complain about Heat, Plant a tree"
>>
>> On Tue, May 22, 2018 at 1:37 PM, Dave Page <[email protected]> wrote:
>>
>>> Hi
>>>
>>> Pivotal's buildbot is showing problems with this patch:
>>>
>>> https://gpdb-dev.bosh.pivotalci.info/teams/pgadmin/
>>> pipelines/pgadmin-patch/jobs/run-linter/builds/66 (linter failed)
>>> https://gpdb-dev.bosh.pivotalci.info/teams/pgadmin/
>>> pipelines/pgadmin-patch/jobs/run-tests/builds/84 (tests failed)
>>>
>>>
>>> On Tue, May 22, 2018 at 7:05 AM, Aditya Toshniwal <aditya.toshniwal@
>>> enterprisedb.com> wrote:
>>>
>>>> Hi Hackers,
>>>>
>>>> PFA patch for RM#3289 where decode error was thrown on querying a
>>>> SQL_ASCII database table. Please note, this problem occurs only on windows.
>>>> Sample insert - insert into test_tab values ('é');
>>>>
>>>> psycopg2 has a encodings dictionary where Postgres Database Encodings
>>>> are mapped to python equivalent. It uses 'ascii' decoder of python to
>>>> decode for SQL_ASCII encoding. If data has characters beyond the limit of
>>>> ascii then it failed. The solution would be to use utf_8 decoder instead of
>>>> ascii. I tried setting the client_encoding using
>>>> set_client_encoding('UTF8') method of a psycopg2 connection but no luck
>>>> (also its not allowed for async connection). I also tried executing "SET
>>>> CLIENT_ENCODING='UTF8'" but it didn't work too.
>>>> So, as in the patch, I had to set encodings dict value directly to
>>>> 'utf_8' and it seems to be working. Please note, the same is added to
>>>> psycopg3 milestones
>>>> https://github.com/psycopg/psycopg2/milestone/4
>>>>
>>>> Also fixed a small glitch for sql editor connection status check.
>>>>
>>>> Kindly review.
>>>>
>>>> Thanks and Regards,
>>>> Aditya Toshniwal
>>>> Software Engineer | EnterpriseDB Software Solutions | Pune
>>>> "Don't Complain about Heat, Plant a tree"
>>>>
>>>
>>>
>>>
>>> --
>>> Dave Page
>>> Blog: http://pgsnake.blogspot.com
>>> Twitter: @pgsnake
>>>
>>> EnterpriseDB UK: http://www.enterprisedb.com
>>> The Enterprise PostgreSQL Company
>>>
>>
>>
^ permalink raw reply [nested|flat] 38+ messages in thread
* Re: [pgAdmin4][RM#3289] Can't query SQL_ASCII database.
@ 2018-05-30 07:06 Aditya Toshniwal <[email protected]>
parent: Aditya Toshniwal <[email protected]>
0 siblings, 1 reply; 38+ messages in thread
From: Aditya Toshniwal @ 2018-05-30 07:06 UTC (permalink / raw)
To: pgadmin-hackers
Hi Hackers,
PFA updated patch after all the permutations, combinations for encoding for
SQL_ASCII database. Also fixed a small glitch for sql editor connection
status check.
Please note, ERROR: invalid byte sequence for encoding "UTF8": 0xe9 0x73 is
a Postgres DB error and not pgAdmin4 error.
You need to change client_encoding to the appropriate. After changing
client_encoding using command - set client_encoding='XYZ', it will give not
give error.
Thanks and Regards,
Aditya Toshniwal
Software Engineer | EnterpriseDB Software Solutions | Pune
"Don't Complain about Heat, Plant a tree"
On Wed, May 23, 2018 at 10:13 AM, Aditya Toshniwal <
[email protected]> wrote:
> Thank you Victoria, Anthony.
>
> Thanks and Regards,
> Aditya Toshniwal
> Software Engineer | EnterpriseDB Software Solutions | Pune
> "Don't Complain about Heat, Plant a tree"
>
> On Tue, May 22, 2018 at 7:15 PM, Victoria Henry <[email protected]> wrote:
>
>> Hi Aditya,
>>
>> We made a minor change to make the patch so the python linter can pass.
>> Attached is the change we made.
>> Everything else looks good.
>>
>> Sincerely,
>>
>> Victoria & Anthony
>>
>> On Tue, May 22, 2018 at 4:46 AM Aditya Toshniwal <
>> [email protected]> wrote:
>>
>>> Hi,
>>>
>>> PFA updated patch. Linter issues are fixed ( we dont have any linter
>>> setup for python :-( )
>>> Regarding test cases, they run successfully on my system and the reason
>>> it failed for pivotal is timeout exception. I am sorry I can't help with
>>> that.
>>>
>>> Traceback (most recent call last):
>>> File "/tmp/build/a453582b/pgadmin-repo/web/pgadmin/feature_tests/keyboard_shortcut_test.py",
>>> line 52, in runTest
>>> self._check_shortcuts()
>>> File "/tmp/build/a453582b/pgadmin-repo/web/pgadmin/feature_tests/keyboard_shortcut_test.py",
>>> line 77, in _check_shortcuts
>>> ") and contains(@class, 'open')]")
>>> File "/root/.pyenv/versions/pgadmin36/lib/python3.6/site-packages
>>> /selenium/webdriver/support/wait.py", line 80, in until
>>> raise TimeoutException(message, screen, stacktrace)
>>> selenium.common.exceptions.TimeoutException: Message:
>>>
>>> Thanks and Regards,
>>> Aditya Toshniwal
>>> Software Engineer | EnterpriseDB Software Solutions | Pune
>>> "Don't Complain about Heat, Plant a tree"
>>>
>>> On Tue, May 22, 2018 at 1:37 PM, Dave Page <[email protected]> wrote:
>>>
>>>> Hi
>>>>
>>>> Pivotal's buildbot is showing problems with this patch:
>>>>
>>>> https://gpdb-dev.bosh.pivotalci.info/teams/pgadmin/pipelines
>>>> /pgadmin-patch/jobs/run-linter/builds/66 (linter failed)
>>>> https://gpdb-dev.bosh.pivotalci.info/teams/pgadmin/pipelines
>>>> /pgadmin-patch/jobs/run-tests/builds/84 (tests failed)
>>>>
>>>>
>>>> On Tue, May 22, 2018 at 7:05 AM, Aditya Toshniwal <
>>>> [email protected]> wrote:
>>>>
>>>>> Hi Hackers,
>>>>>
>>>>> PFA patch for RM#3289 where decode error was thrown on querying a
>>>>> SQL_ASCII database table. Please note, this problem occurs only on windows.
>>>>> Sample insert - insert into test_tab values ('é');
>>>>>
>>>>> psycopg2 has a encodings dictionary where Postgres Database Encodings
>>>>> are mapped to python equivalent. It uses 'ascii' decoder of python to
>>>>> decode for SQL_ASCII encoding. If data has characters beyond the limit of
>>>>> ascii then it failed. The solution would be to use utf_8 decoder instead of
>>>>> ascii. I tried setting the client_encoding using
>>>>> set_client_encoding('UTF8') method of a psycopg2 connection but no luck
>>>>> (also its not allowed for async connection). I also tried executing "SET
>>>>> CLIENT_ENCODING='UTF8'" but it didn't work too.
>>>>> So, as in the patch, I had to set encodings dict value directly to
>>>>> 'utf_8' and it seems to be working. Please note, the same is added to
>>>>> psycopg3 milestones
>>>>> https://github.com/psycopg/psycopg2/milestone/4
>>>>>
>>>>> Also fixed a small glitch for sql editor connection status check.
>>>>>
>>>>> Kindly review.
>>>>>
>>>>> Thanks and Regards,
>>>>> Aditya Toshniwal
>>>>> Software Engineer | EnterpriseDB Software Solutions | Pune
>>>>> "Don't Complain about Heat, Plant a tree"
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Dave Page
>>>> Blog: http://pgsnake.blogspot.com
>>>> Twitter: @pgsnake
>>>>
>>>> EnterpriseDB UK: http://www.enterprisedb.com
>>>> The Enterprise PostgreSQL Company
>>>>
>>>
>>>
>
Attachments:
[image/png] image.png (8.7K, 3-image.png)
download | view image
[image/png] image.png (9.0K, 4-image.png)
download | view image
[application/octet-stream] RM3289.patch (2.4K, 5-RM3289.patch)
download | inline diff:
diff --git a/web/pgadmin/tools/sqleditor/__init__.py b/web/pgadmin/tools/sqleditor/__init__.py
index c72505a4..ead100e7 100644
--- a/web/pgadmin/tools/sqleditor/__init__.py
+++ b/web/pgadmin/tools/sqleditor/__init__.py
@@ -1476,14 +1476,19 @@ def query_tool_status(trans_id):
if conn and trans_obj and session_obj:
status = conn.transaction_status()
- return make_json_response(
- data={
- 'status': status,
- 'message': gettext(
- CONNECTION_STATUS_MESSAGE_MAPPING.get(status)
- )
- }
- )
+ if status is not None:
+ return make_json_response(
+ data={
+ 'status': status,
+ 'message': gettext(
+ CONNECTION_STATUS_MESSAGE_MAPPING.get(status)
+ )
+ }
+ )
+ else:
+ return internal_server_error(
+ errormsg=gettext("Transaction status check failed.")
+ )
else:
return internal_server_error(
errormsg=gettext("Transaction status check failed.")
diff --git a/web/pgadmin/utils/driver/psycopg2/connection.py b/web/pgadmin/utils/driver/psycopg2/connection.py
index 315631c0..ac85f9dc 100644
--- a/web/pgadmin/utils/driver/psycopg2/connection.py
+++ b/web/pgadmin/utils/driver/psycopg2/connection.py
@@ -50,6 +50,12 @@ else:
_ = gettext
+# Replace default ascii encoder with unicode-escape
+# which translates characters to unicode format.
+# Escape special characters to ASCII based on unicode
+encodings['SQL_ASCII'] = 'unicode-escape'
+encodings['SQLASCII'] = 'unicode-escape'
+
# Register global type caster which will be applicable to all connections.
register_global_typecasters()
diff --git a/web/pgadmin/utils/driver/psycopg2/typecast.py b/web/pgadmin/utils/driver/psycopg2/typecast.py
index f1366049..ccebb2f9 100644
--- a/web/pgadmin/utils/driver/psycopg2/typecast.py
+++ b/web/pgadmin/utils/driver/psycopg2/typecast.py
@@ -164,7 +164,7 @@ def register_global_typecasters():
def register_string_typecasters(connection):
- if connection.encoding != 'UTF8':
+ if connection.encoding not in ('UTF8', 'SQLASCII', 'SQL_ASCII'):
# In python3 when database encoding is other than utf-8 and client
# encoding is set to UNICODE then we need to map data from database
# encoding to utf-8.
^ permalink raw reply [nested|flat] 38+ messages in thread
* Re: [pgAdmin4][RM#3289] Can't query SQL_ASCII database.
@ 2018-05-30 20:36 Joao De Almeida Pereira <[email protected]>
parent: Aditya Toshniwal <[email protected]>
0 siblings, 1 reply; 38+ messages in thread
From: Joao De Almeida Pereira @ 2018-05-30 20:36 UTC (permalink / raw)
To: Aditya Toshniwal <[email protected]>; +Cc: pgadmin-hackers
Hello Aditya,
It looks ok and it passes CI.
We have some recommendations:
- These look like 2 different changes so they should be in separated commits
- Do we have test coverage for the bug that you are talking about? If not
we should, to ensure this problem does not happen again in a future change.
Thanks
Victoria && Joao
On Wed, May 30, 2018 at 3:06 AM Aditya Toshniwal <
[email protected]> wrote:
> Hi Hackers,
>
> PFA updated patch after all the permutations, combinations for encoding
> for SQL_ASCII database. Also fixed a small glitch for sql editor
> connection status check.
>
> Please note, ERROR: invalid byte sequence for encoding "UTF8": 0xe9 0x73
> is a Postgres DB error and not pgAdmin4 error.
>
>
>
> You need to change client_encoding to the appropriate. After changing
> client_encoding using command - set client_encoding='XYZ', it will give not
> give error.
>
>
>
>
>
>
>
>
> Thanks and Regards,
> Aditya Toshniwal
> Software Engineer | EnterpriseDB Software Solutions | Pune
> "Don't Complain about Heat, Plant a tree"
>
> On Wed, May 23, 2018 at 10:13 AM, Aditya Toshniwal <
> [email protected]> wrote:
>
>> Thank you Victoria, Anthony.
>>
>> Thanks and Regards,
>> Aditya Toshniwal
>> Software Engineer | EnterpriseDB Software Solutions | Pune
>> "Don't Complain about Heat, Plant a tree"
>>
>> On Tue, May 22, 2018 at 7:15 PM, Victoria Henry <[email protected]>
>> wrote:
>>
>>> Hi Aditya,
>>>
>>> We made a minor change to make the patch so the python linter can pass.
>>> Attached is the change we made.
>>> Everything else looks good.
>>>
>>> Sincerely,
>>>
>>> Victoria & Anthony
>>>
>>> On Tue, May 22, 2018 at 4:46 AM Aditya Toshniwal <
>>> [email protected]> wrote:
>>>
>>>> Hi,
>>>>
>>>> PFA updated patch. Linter issues are fixed ( we dont have any linter
>>>> setup for python :-( )
>>>> Regarding test cases, they run successfully on my system and the reason
>>>> it failed for pivotal is timeout exception. I am sorry I can't help with
>>>> that.
>>>>
>>>> Traceback (most recent call last):
>>>> File
>>>> "/tmp/build/a453582b/pgadmin-repo/web/pgadmin/feature_tests/keyboard_shortcut_test.py",
>>>> line 52, in runTest
>>>> self._check_shortcuts()
>>>> File
>>>> "/tmp/build/a453582b/pgadmin-repo/web/pgadmin/feature_tests/keyboard_shortcut_test.py",
>>>> line 77, in _check_shortcuts
>>>> ") and contains(@class, 'open')]")
>>>> File
>>>> "/root/.pyenv/versions/pgadmin36/lib/python3.6/site-packages/selenium/webdriver/support/wait.py",
>>>> line 80, in until
>>>> raise TimeoutException(message, screen, stacktrace)
>>>> selenium.common.exceptions.TimeoutException: Message:
>>>>
>>>> Thanks and Regards,
>>>> Aditya Toshniwal
>>>> Software Engineer | EnterpriseDB Software Solutions | Pune
>>>> "Don't Complain about Heat, Plant a tree"
>>>>
>>>> On Tue, May 22, 2018 at 1:37 PM, Dave Page <[email protected]> wrote:
>>>>
>>>>> Hi
>>>>>
>>>>> Pivotal's buildbot is showing problems with this patch:
>>>>>
>>>>>
>>>>> https://gpdb-dev.bosh.pivotalci.info/teams/pgadmin/pipelines/pgadmin-patch/jobs/run-linter/builds/66
>>>>> (linter failed)
>>>>>
>>>>> https://gpdb-dev.bosh.pivotalci.info/teams/pgadmin/pipelines/pgadmin-patch/jobs/run-tests/builds/84
>>>>> (tests failed)
>>>>>
>>>>>
>>>>> On Tue, May 22, 2018 at 7:05 AM, Aditya Toshniwal <
>>>>> [email protected]> wrote:
>>>>>
>>>>>> Hi Hackers,
>>>>>>
>>>>>> PFA patch for RM#3289 where decode error was thrown on querying a
>>>>>> SQL_ASCII database table. Please note, this problem occurs only on windows.
>>>>>> Sample insert - insert into test_tab values ('é');
>>>>>>
>>>>>> psycopg2 has a encodings dictionary where Postgres Database Encodings
>>>>>> are mapped to python equivalent. It uses 'ascii' decoder of python to
>>>>>> decode for SQL_ASCII encoding. If data has characters beyond the limit of
>>>>>> ascii then it failed. The solution would be to use utf_8 decoder instead of
>>>>>> ascii. I tried setting the client_encoding using
>>>>>> set_client_encoding('UTF8') method of a psycopg2 connection but no luck
>>>>>> (also its not allowed for async connection). I also tried executing "SET
>>>>>> CLIENT_ENCODING='UTF8'" but it didn't work too.
>>>>>> So, as in the patch, I had to set encodings dict value directly to
>>>>>> 'utf_8' and it seems to be working. Please note, the same is added to
>>>>>> psycopg3 milestones
>>>>>> https://github.com/psycopg/psycopg2/milestone/4
>>>>>>
>>>>>> Also fixed a small glitch for sql editor connection status check.
>>>>>>
>>>>>> Kindly review.
>>>>>>
>>>>>> Thanks and Regards,
>>>>>> Aditya Toshniwal
>>>>>> Software Engineer | EnterpriseDB Software Solutions | Pune
>>>>>> "Don't Complain about Heat, Plant a tree"
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Dave Page
>>>>> Blog: http://pgsnake.blogspot.com
>>>>> Twitter: @pgsnake
>>>>>
>>>>> EnterpriseDB UK: http://www.enterprisedb.com
>>>>> The Enterprise PostgreSQL Company
>>>>>
>>>>
>>>>
>>
>
Attachments:
[image/png] image.png (8.7K, 3-image.png)
download | view image
[image/png] image.png (9.0K, 4-image.png)
download | view image
^ permalink raw reply [nested|flat] 38+ messages in thread
* Re: [pgAdmin4][RM#3289] Can't query SQL_ASCII database.
@ 2018-05-31 05:20 Aditya Toshniwal <[email protected]>
parent: Joao De Almeida Pereira <[email protected]>
0 siblings, 1 reply; 38+ messages in thread
From: Aditya Toshniwal @ 2018-05-31 05:20 UTC (permalink / raw)
To: Joao De Almeida Pereira <[email protected]>; +Cc: pgadmin-hackers
Hi Victoria/Joao,
On Thu, May 31, 2018 at 2:06 AM, Joao De Almeida Pereira <
[email protected]> wrote:
> Hello Aditya,
>
> It looks ok and it passes CI.
>
> We have some recommendations:
> - These look like 2 different changes so they should be in separated
> commits
>
If you are talking of set client_encoding, then its not a bug. Its a choice
given to Postgres DB user to change the encoding of the characters.
Postgres will translate characters from Server Encoding to Client Encoding,
and will throw error like mentioned previously. This link will help better
- https://www.postgresql.org/docs/10/static/multibyte.html
The actual bug was, even after changing the client encoding to SQL_ASCII,
pgAdmin4 was not able to show the output as it was failing in encoding by
psycopg2. The patch is for resolving that.
> - Do we have test coverage for the bug that you are talking about? If not
> we should, to ensure this problem does not happen again in a future change.
>
It is not possible adding test cases for encoding related stuff, as
Postgres support a lot many different types of encoding and conversions
(refer above link)
>
> Thanks
> Victoria && Joao
>
> On Wed, May 30, 2018 at 3:06 AM Aditya Toshniwal <aditya.toshniwal@
> enterprisedb.com> wrote:
>
>> Hi Hackers,
>>
>> PFA updated patch after all the permutations, combinations for encoding
>> for SQL_ASCII database. Also fixed a small glitch for sql editor
>> connection status check.
>>
>> Please note, ERROR: invalid byte sequence for encoding "UTF8": 0xe9 0x73
>> is a Postgres DB error and not pgAdmin4 error.
>>
>> <Image Deleted>
>>
>> You need to change client_encoding to the appropriate. After changing
>> client_encoding using command - set client_encoding='XYZ', it will give not
>> give error.
>>
>> <Image Deleted>
>>
>> Thanks and Regards,
>> Aditya Toshniwal
>> Software Engineer | EnterpriseDB Software Solutions | Pune
>> "Don't Complain about Heat, Plant a tree"
>>
>> On Wed, May 23, 2018 at 10:13 AM, Aditya Toshniwal <aditya.toshniwal@
>> enterprisedb.com> wrote:
>>
>>> Thank you Victoria, Anthony.
>>>
>>> Thanks and Regards,
>>> Aditya Toshniwal
>>> Software Engineer | EnterpriseDB Software Solutions | Pune
>>> "Don't Complain about Heat, Plant a tree"
>>>
>>> On Tue, May 22, 2018 at 7:15 PM, Victoria Henry <[email protected]>
>>> wrote:
>>>
>>>> Hi Aditya,
>>>>
>>>> We made a minor change to make the patch so the python linter can
>>>> pass. Attached is the change we made.
>>>> Everything else looks good.
>>>>
>>>> Sincerely,
>>>>
>>>> Victoria & Anthony
>>>>
>>>> On Tue, May 22, 2018 at 4:46 AM Aditya Toshniwal <aditya.toshniwal@
>>>> enterprisedb.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> PFA updated patch. Linter issues are fixed ( we dont have any linter
>>>>> setup for python :-( )
>>>>> Regarding test cases, they run successfully on my system and the
>>>>> reason it failed for pivotal is timeout exception. I am sorry I can't help
>>>>> with that.
>>>>>
>>>>> Traceback (most recent call last):
>>>>> File "/tmp/build/a453582b/pgadmin-repo/web/pgadmin/feature_
>>>>> tests/keyboard_shortcut_test.py", line 52, in runTest
>>>>> self._check_shortcuts()
>>>>> File "/tmp/build/a453582b/pgadmin-repo/web/pgadmin/feature_
>>>>> tests/keyboard_shortcut_test.py", line 77, in _check_shortcuts
>>>>> ") and contains(@class, 'open')]")
>>>>> File "/root/.pyenv/versions/pgadmin36/lib/python3.6/site-
>>>>> packages/selenium/webdriver/support/wait.py", line 80, in until
>>>>> raise TimeoutException(message, screen, stacktrace)
>>>>> selenium.common.exceptions.TimeoutException: Message:
>>>>>
>>>>> Thanks and Regards,
>>>>> Aditya Toshniwal
>>>>> Software Engineer | EnterpriseDB Software Solutions | Pune
>>>>> "Don't Complain about Heat, Plant a tree"
>>>>>
>>>>> On Tue, May 22, 2018 at 1:37 PM, Dave Page <[email protected]> wrote:
>>>>>
>>>>>> Hi
>>>>>>
>>>>>> Pivotal's buildbot is showing problems with this patch:
>>>>>>
>>>>>> https://gpdb-dev.bosh.pivotalci.info/teams/pgadmin/
>>>>>> pipelines/pgadmin-patch/jobs/run-linter/builds/66 (linter failed)
>>>>>> https://gpdb-dev.bosh.pivotalci.info/teams/pgadmin/
>>>>>> pipelines/pgadmin-patch/jobs/run-tests/builds/84 (tests failed)
>>>>>>
>>>>>>
>>>>>> On Tue, May 22, 2018 at 7:05 AM, Aditya Toshniwal <aditya.toshniwal@
>>>>>> enterprisedb.com> wrote:
>>>>>>
>>>>>>> Hi Hackers,
>>>>>>>
>>>>>>> PFA patch for RM#3289 where decode error was thrown on querying a
>>>>>>> SQL_ASCII database table. Please note, this problem occurs only on windows.
>>>>>>> Sample insert - insert into test_tab values ('é');
>>>>>>>
>>>>>>> psycopg2 has a encodings dictionary where Postgres Database
>>>>>>> Encodings are mapped to python equivalent. It uses 'ascii' decoder of
>>>>>>> python to decode for SQL_ASCII encoding. If data has characters beyond the
>>>>>>> limit of ascii then it failed. The solution would be to use utf_8 decoder
>>>>>>> instead of ascii. I tried setting the client_encoding using
>>>>>>> set_client_encoding('UTF8') method of a psycopg2 connection but no luck
>>>>>>> (also its not allowed for async connection). I also tried executing "SET
>>>>>>> CLIENT_ENCODING='UTF8'" but it didn't work too.
>>>>>>> So, as in the patch, I had to set encodings dict value directly to
>>>>>>> 'utf_8' and it seems to be working. Please note, the same is added to
>>>>>>> psycopg3 milestones
>>>>>>> https://github.com/psycopg/psycopg2/milestone/4
>>>>>>>
>>>>>>> Also fixed a small glitch for sql editor connection status check.
>>>>>>>
>>>>>>> Kindly review.
>>>>>>>
>>>>>>> Thanks and Regards,
>>>>>>> Aditya Toshniwal
>>>>>>> Software Engineer | EnterpriseDB Software Solutions | Pune
>>>>>>> "Don't Complain about Heat, Plant a tree"
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Dave Page
>>>>>> Blog: http://pgsnake.blogspot.com
>>>>>> Twitter: @pgsnake
>>>>>>
>>>>>> EnterpriseDB UK: http://www.enterprisedb.com
>>>>>> The Enterprise PostgreSQL Company
>>>>>>
>>>>>
>>>>>
>>>
>>
^ permalink raw reply [nested|flat] 38+ messages in thread
* Re: [pgAdmin4][RM#3289] Can't query SQL_ASCII database.
@ 2018-05-31 13:12 Dave Page <[email protected]>
parent: Aditya Toshniwal <[email protected]>
0 siblings, 1 reply; 38+ messages in thread
From: Dave Page @ 2018-05-31 13:12 UTC (permalink / raw)
To: Aditya Toshniwal <[email protected]>; +Cc: Joao De Almeida Pereira <[email protected]>; pgadmin-hackers
Hi
On Thu, May 31, 2018 at 1:20 AM, Aditya Toshniwal <
[email protected]> wrote:
> Hi Victoria/Joao,
>
> On Thu, May 31, 2018 at 2:06 AM, Joao De Almeida Pereira <
> [email protected]> wrote:
>
>> Hello Aditya,
>>
>> It looks ok and it passes CI.
>>
>> We have some recommendations:
>> - These look like 2 different changes so they should be in separated
>> commits
>>
>
> If you are talking of set client_encoding, then its not a bug. Its a
> choice given to Postgres DB user to change the encoding of the characters.
> Postgres will translate characters from Server Encoding to Client Encoding,
> and will throw error like mentioned previously. This link will help better
> - https://www.postgresql.org/docs/10/static/multibyte.html
> The actual bug was, even after changing the client encoding to SQL_ASCII,
> pgAdmin4 was not able to show the output as it was failing in encoding by
> psycopg2. The patch is for resolving that.
>
>
>> - Do we have test coverage for the bug that you are talking about? If not
>> we should, to ensure this problem does not happen again in a future change.
>>
>
> It is not possible adding test cases for encoding related stuff, as
> Postgres support a lot many different types of encoding and conversions
> (refer above link)
>
I was going to ask the same thing. Per
https://www.postgresql.org/docs/10/static/multibyte.html#id-1.6.10.5.7,
every characterset except SQL_ASCII can be converted to UTF-8, so we only
need to test that UTF-8 and some other charactersets besides SQL_ASCII
work, and then separately that SQL_ASCII with characters known not to be in
UTF-8 work.
>
>> Thanks
>> Victoria && Joao
>>
>> On Wed, May 30, 2018 at 3:06 AM Aditya Toshniwal <
>> [email protected]> wrote:
>>
>>> Hi Hackers,
>>>
>>> PFA updated patch after all the permutations, combinations for encoding
>>> for SQL_ASCII database. Also fixed a small glitch for sql editor
>>> connection status check.
>>>
>>> Please note, ERROR: invalid byte sequence for encoding "UTF8": 0xe9 0x73
>>> is a Postgres DB error and not pgAdmin4 error.
>>>
>>> <Image Deleted>
>>>
>>> You need to change client_encoding to the appropriate. After changing
>>> client_encoding using command - set client_encoding='XYZ', it will give not
>>> give error.
>>>
>>> <Image Deleted>
>>>
>>> Thanks and Regards,
>>> Aditya Toshniwal
>>> Software Engineer | EnterpriseDB Software Solutions | Pune
>>> "Don't Complain about Heat, Plant a tree"
>>>
>>> On Wed, May 23, 2018 at 10:13 AM, Aditya Toshniwal <
>>> [email protected]> wrote:
>>>
>>>> Thank you Victoria, Anthony.
>>>>
>>>> Thanks and Regards,
>>>> Aditya Toshniwal
>>>> Software Engineer | EnterpriseDB Software Solutions | Pune
>>>> "Don't Complain about Heat, Plant a tree"
>>>>
>>>> On Tue, May 22, 2018 at 7:15 PM, Victoria Henry <[email protected]>
>>>> wrote:
>>>>
>>>>> Hi Aditya,
>>>>>
>>>>> We made a minor change to make the patch so the python linter can
>>>>> pass. Attached is the change we made.
>>>>> Everything else looks good.
>>>>>
>>>>> Sincerely,
>>>>>
>>>>> Victoria & Anthony
>>>>>
>>>>> On Tue, May 22, 2018 at 4:46 AM Aditya Toshniwal <
>>>>> [email protected]> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> PFA updated patch. Linter issues are fixed ( we dont have any linter
>>>>>> setup for python :-( )
>>>>>> Regarding test cases, they run successfully on my system and the
>>>>>> reason it failed for pivotal is timeout exception. I am sorry I can't help
>>>>>> with that.
>>>>>>
>>>>>> Traceback (most recent call last):
>>>>>> File "/tmp/build/a453582b/pgadmin-repo/web/pgadmin/feature_tests/keyboard_shortcut_test.py",
>>>>>> line 52, in runTest
>>>>>> self._check_shortcuts()
>>>>>> File "/tmp/build/a453582b/pgadmin-repo/web/pgadmin/feature_tests/keyboard_shortcut_test.py",
>>>>>> line 77, in _check_shortcuts
>>>>>> ") and contains(@class, 'open')]")
>>>>>> File "/root/.pyenv/versions/pgadmin36/lib/python3.6/site-packages
>>>>>> /selenium/webdriver/support/wait.py", line 80, in until
>>>>>> raise TimeoutException(message, screen, stacktrace)
>>>>>> selenium.common.exceptions.TimeoutException: Message:
>>>>>>
>>>>>> Thanks and Regards,
>>>>>> Aditya Toshniwal
>>>>>> Software Engineer | EnterpriseDB Software Solutions | Pune
>>>>>> "Don't Complain about Heat, Plant a tree"
>>>>>>
>>>>>> On Tue, May 22, 2018 at 1:37 PM, Dave Page <[email protected]> wrote:
>>>>>>
>>>>>>> Hi
>>>>>>>
>>>>>>> Pivotal's buildbot is showing problems with this patch:
>>>>>>>
>>>>>>> https://gpdb-dev.bosh.pivotalci.info/teams/pgadmin/pipelines
>>>>>>> /pgadmin-patch/jobs/run-linter/builds/66 (linter failed)
>>>>>>> https://gpdb-dev.bosh.pivotalci.info/teams/pgadmin/pipelines
>>>>>>> /pgadmin-patch/jobs/run-tests/builds/84 (tests failed)
>>>>>>>
>>>>>>>
>>>>>>> On Tue, May 22, 2018 at 7:05 AM, Aditya Toshniwal <
>>>>>>> [email protected]> wrote:
>>>>>>>
>>>>>>>> Hi Hackers,
>>>>>>>>
>>>>>>>> PFA patch for RM#3289 where decode error was thrown on querying a
>>>>>>>> SQL_ASCII database table. Please note, this problem occurs only on windows.
>>>>>>>> Sample insert - insert into test_tab values ('é');
>>>>>>>>
>>>>>>>> psycopg2 has a encodings dictionary where Postgres Database
>>>>>>>> Encodings are mapped to python equivalent. It uses 'ascii' decoder of
>>>>>>>> python to decode for SQL_ASCII encoding. If data has characters beyond the
>>>>>>>> limit of ascii then it failed. The solution would be to use utf_8 decoder
>>>>>>>> instead of ascii. I tried setting the client_encoding using
>>>>>>>> set_client_encoding('UTF8') method of a psycopg2 connection but no luck
>>>>>>>> (also its not allowed for async connection). I also tried executing "SET
>>>>>>>> CLIENT_ENCODING='UTF8'" but it didn't work too.
>>>>>>>> So, as in the patch, I had to set encodings dict value directly to
>>>>>>>> 'utf_8' and it seems to be working. Please note, the same is added to
>>>>>>>> psycopg3 milestones
>>>>>>>> https://github.com/psycopg/psycopg2/milestone/4
>>>>>>>>
>>>>>>>> Also fixed a small glitch for sql editor connection status check.
>>>>>>>>
>>>>>>>> Kindly review.
>>>>>>>>
>>>>>>>> Thanks and Regards,
>>>>>>>> Aditya Toshniwal
>>>>>>>> Software Engineer | EnterpriseDB Software Solutions | Pune
>>>>>>>> "Don't Complain about Heat, Plant a tree"
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Dave Page
>>>>>>> Blog: http://pgsnake.blogspot.com
>>>>>>> Twitter: @pgsnake
>>>>>>>
>>>>>>> EnterpriseDB UK: http://www.enterprisedb.com
>>>>>>> The Enterprise PostgreSQL Company
>>>>>>>
>>>>>>
>>>>>>
>>>>
>>>
>
--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake
EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 38+ messages in thread
* Re: [pgAdmin4][RM#3289] Can't query SQL_ASCII database.
@ 2018-06-04 05:23 Aditya Toshniwal <[email protected]>
parent: Dave Page <[email protected]>
0 siblings, 1 reply; 38+ messages in thread
From: Aditya Toshniwal @ 2018-06-04 05:23 UTC (permalink / raw)
To: Dave Page <[email protected]>; +Cc: Joao De Almeida Pereira <[email protected]>; pgadmin-hackers
Hi Hackers,
PFA the updated patch on this. I have tried to add test cases to check for
different encoding database. In the test run, it will create a database,
fire a query for a string, check if we get the output and drops the
database.
Kindly review.
Thanks and Regards,
Aditya Toshniwal
Software Engineer | EnterpriseDB Software Solutions | Pune
"Don't Complain about Heat, Plant a tree"
On Thu, May 31, 2018 at 6:42 PM, Dave Page <[email protected]> wrote:
> Hi
>
> On Thu, May 31, 2018 at 1:20 AM, Aditya Toshniwal <aditya.toshniwal@
> enterprisedb.com> wrote:
>
>> Hi Victoria/Joao,
>>
>> On Thu, May 31, 2018 at 2:06 AM, Joao De Almeida Pereira <
>> [email protected]> wrote:
>>
>>> Hello Aditya,
>>>
>>> It looks ok and it passes CI.
>>>
>>> We have some recommendations:
>>> - These look like 2 different changes so they should be in separated
>>> commits
>>>
>>
>> If you are talking of set client_encoding, then its not a bug. Its a
>> choice given to Postgres DB user to change the encoding of the characters.
>> Postgres will translate characters from Server Encoding to Client Encoding,
>> and will throw error like mentioned previously. This link will help better
>> - https://www.postgresql.org/docs/10/static/multibyte.html
>> The actual bug was, even after changing the client encoding to SQL_ASCII,
>> pgAdmin4 was not able to show the output as it was failing in encoding by
>> psycopg2. The patch is for resolving that.
>>
>>
>>> - Do we have test coverage for the bug that you are talking about? If
>>> not we should, to ensure this problem does not happen again in a future
>>> change.
>>>
>>
>> It is not possible adding test cases for encoding related stuff, as
>> Postgres support a lot many different types of encoding and conversions
>> (refer above link)
>>
>
> I was going to ask the same thing. Per https://www.postgresql.
> org/docs/10/static/multibyte.html#id-1.6.10.5.7, every characterset
> except SQL_ASCII can be converted to UTF-8, so we only need to test that
> UTF-8 and some other charactersets besides SQL_ASCII work, and then
> separately that SQL_ASCII with characters known not to be in UTF-8 work.
>
>
>>
>>> Thanks
>>> Victoria && Joao
>>>
>>> On Wed, May 30, 2018 at 3:06 AM Aditya Toshniwal <
>>> [email protected]> wrote:
>>>
>>>> Hi Hackers,
>>>>
>>>> PFA updated patch after all the permutations, combinations for encoding
>>>> for SQL_ASCII database. Also fixed a small glitch for sql editor
>>>> connection status check.
>>>>
>>>> Please note, ERROR: invalid byte sequence for encoding "UTF8": 0xe9
>>>> 0x73 is a Postgres DB error and not pgAdmin4 error.
>>>>
>>>> <Image Deleted>
>>>>
>>>> You need to change client_encoding to the appropriate. After changing
>>>> client_encoding using command - set client_encoding='XYZ', it will give not
>>>> give error.
>>>>
>>>> <Image Deleted>
>>>>
>>>> Thanks and Regards,
>>>> Aditya Toshniwal
>>>> Software Engineer | EnterpriseDB Software Solutions | Pune
>>>> "Don't Complain about Heat, Plant a tree"
>>>>
>>>> On Wed, May 23, 2018 at 10:13 AM, Aditya Toshniwal <
>>>> [email protected]> wrote:
>>>>
>>>>> Thank you Victoria, Anthony.
>>>>>
>>>>> Thanks and Regards,
>>>>> Aditya Toshniwal
>>>>> Software Engineer | EnterpriseDB Software Solutions | Pune
>>>>> "Don't Complain about Heat, Plant a tree"
>>>>>
>>>>> On Tue, May 22, 2018 at 7:15 PM, Victoria Henry <[email protected]>
>>>>> wrote:
>>>>>
>>>>>> Hi Aditya,
>>>>>>
>>>>>> We made a minor change to make the patch so the python linter can
>>>>>> pass. Attached is the change we made.
>>>>>> Everything else looks good.
>>>>>>
>>>>>> Sincerely,
>>>>>>
>>>>>> Victoria & Anthony
>>>>>>
>>>>>> On Tue, May 22, 2018 at 4:46 AM Aditya Toshniwal <
>>>>>> [email protected]> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> PFA updated patch. Linter issues are fixed ( we dont have any linter
>>>>>>> setup for python :-( )
>>>>>>> Regarding test cases, they run successfully on my system and the
>>>>>>> reason it failed for pivotal is timeout exception. I am sorry I can't help
>>>>>>> with that.
>>>>>>>
>>>>>>> Traceback (most recent call last):
>>>>>>> File "/tmp/build/a453582b/pgadmin-repo/web/pgadmin/feature_tests/keyboard_shortcut_test.py",
>>>>>>> line 52, in runTest
>>>>>>> self._check_shortcuts()
>>>>>>> File "/tmp/build/a453582b/pgadmin-repo/web/pgadmin/feature_tests/keyboard_shortcut_test.py",
>>>>>>> line 77, in _check_shortcuts
>>>>>>> ") and contains(@class, 'open')]")
>>>>>>> File "/root/.pyenv/versions/pgadmin36/lib/python3.6/site-packages
>>>>>>> /selenium/webdriver/support/wait.py", line 80, in until
>>>>>>> raise TimeoutException(message, screen, stacktrace)
>>>>>>> selenium.common.exceptions.TimeoutException: Message:
>>>>>>>
>>>>>>> Thanks and Regards,
>>>>>>> Aditya Toshniwal
>>>>>>> Software Engineer | EnterpriseDB Software Solutions | Pune
>>>>>>> "Don't Complain about Heat, Plant a tree"
>>>>>>>
>>>>>>> On Tue, May 22, 2018 at 1:37 PM, Dave Page <[email protected]>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi
>>>>>>>>
>>>>>>>> Pivotal's buildbot is showing problems with this patch:
>>>>>>>>
>>>>>>>> https://gpdb-dev.bosh.pivotalci.info/teams/pgadmin/pipelines
>>>>>>>> /pgadmin-patch/jobs/run-linter/builds/66 (linter failed)
>>>>>>>> https://gpdb-dev.bosh.pivotalci.info/teams/pgadmin/pipelines
>>>>>>>> /pgadmin-patch/jobs/run-tests/builds/84 (tests failed)
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, May 22, 2018 at 7:05 AM, Aditya Toshniwal <
>>>>>>>> [email protected]> wrote:
>>>>>>>>
>>>>>>>>> Hi Hackers,
>>>>>>>>>
>>>>>>>>> PFA patch for RM#3289 where decode error was thrown on querying a
>>>>>>>>> SQL_ASCII database table. Please note, this problem occurs only on windows.
>>>>>>>>> Sample insert - insert into test_tab values ('é');
>>>>>>>>>
>>>>>>>>> psycopg2 has a encodings dictionary where Postgres Database
>>>>>>>>> Encodings are mapped to python equivalent. It uses 'ascii' decoder of
>>>>>>>>> python to decode for SQL_ASCII encoding. If data has characters beyond the
>>>>>>>>> limit of ascii then it failed. The solution would be to use utf_8 decoder
>>>>>>>>> instead of ascii. I tried setting the client_encoding using
>>>>>>>>> set_client_encoding('UTF8') method of a psycopg2 connection but no luck
>>>>>>>>> (also its not allowed for async connection). I also tried executing "SET
>>>>>>>>> CLIENT_ENCODING='UTF8'" but it didn't work too.
>>>>>>>>> So, as in the patch, I had to set encodings dict value directly to
>>>>>>>>> 'utf_8' and it seems to be working. Please note, the same is added to
>>>>>>>>> psycopg3 milestones
>>>>>>>>> https://github.com/psycopg/psycopg2/milestone/4
>>>>>>>>>
>>>>>>>>> Also fixed a small glitch for sql editor connection status check.
>>>>>>>>>
>>>>>>>>> Kindly review.
>>>>>>>>>
>>>>>>>>> Thanks and Regards,
>>>>>>>>> Aditya Toshniwal
>>>>>>>>> Software Engineer | EnterpriseDB Software Solutions | Pune
>>>>>>>>> "Don't Complain about Heat, Plant a tree"
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Dave Page
>>>>>>>> Blog: http://pgsnake.blogspot.com
>>>>>>>> Twitter: @pgsnake
>>>>>>>>
>>>>>>>> EnterpriseDB UK: http://www.enterprisedb.com
>>>>>>>> The Enterprise PostgreSQL Company
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>
>>>>
>>
>
>
> --
> Dave Page
> Blog: http://pgsnake.blogspot.com
> Twitter: @pgsnake
>
> EnterpriseDB UK: http://www.enterprisedb.com
> The Enterprise PostgreSQL Company
>
Attachments:
[text/x-patch] RM3289.patch (4.0K, 3-RM3289.patch)
download | inline diff:
diff --git a/web/pgadmin/tools/sqleditor/__init__.py b/web/pgadmin/tools/sqleditor/__init__.py
index 013e4dc..a9460dd 100644
--- a/web/pgadmin/tools/sqleditor/__init__.py
+++ b/web/pgadmin/tools/sqleditor/__init__.py
@@ -1480,19 +1480,24 @@ def query_tool_status(trans_id):
if conn and trans_obj and session_obj:
status = conn.transaction_status()
- # Check for the asynchronous notifies statements.
- conn.check_notifies(True)
- notifies = conn.get_notifies()
+ if status is not None:
+ # Check for the asynchronous notifies statements.
+ conn.check_notifies(True)
+ notifies = conn.get_notifies()
- return make_json_response(
- data={
- 'status': status,
- 'message': gettext(
- CONNECTION_STATUS_MESSAGE_MAPPING.get(status),
- ),
- 'notifies': notifies
- }
- )
+ return make_json_response(
+ data={
+ 'status': status,
+ 'message': gettext(
+ CONNECTION_STATUS_MESSAGE_MAPPING.get(status),
+ ),
+ 'notifies': notifies
+ }
+ )
+ else:
+ return internal_server_error(
+ errormsg=gettext("Transaction status check failed.")
+ )
else:
return internal_server_error(
errormsg=gettext("Transaction status check failed.")
diff --git a/web/pgadmin/utils/driver/psycopg2/connection.py b/web/pgadmin/utils/driver/psycopg2/connection.py
index cfd161a..e8ca886 100644
--- a/web/pgadmin/utils/driver/psycopg2/connection.py
+++ b/web/pgadmin/utils/driver/psycopg2/connection.py
@@ -51,6 +51,12 @@ else:
_ = gettext
+# Replace default ascii encoder with unicode-escape
+# which translates characters to unicode format.
+# Escape special characters to ASCII based on unicode
+encodings['SQL_ASCII'] = 'unicode-escape'
+encodings['SQLASCII'] = 'unicode-escape'
+
# Register global type caster which will be applicable to all connections.
register_global_typecasters()
diff --git a/web/pgadmin/utils/driver/psycopg2/typecast.py b/web/pgadmin/utils/driver/psycopg2/typecast.py
index f136604..ccebb2f 100644
--- a/web/pgadmin/utils/driver/psycopg2/typecast.py
+++ b/web/pgadmin/utils/driver/psycopg2/typecast.py
@@ -164,7 +164,7 @@ def register_global_typecasters():
def register_string_typecasters(connection):
- if connection.encoding != 'UTF8':
+ if connection.encoding not in ('UTF8', 'SQLASCII', 'SQL_ASCII'):
# In python3 when database encoding is other than utf-8 and client
# encoding is set to UNICODE then we need to map data from database
# encoding to utf-8.
diff --git a/web/regression/python_test_utils/test_utils.py b/web/regression/python_test_utils/test_utils.py
index 3e517b6..163af10 100644
--- a/web/regression/python_test_utils/test_utils.py
+++ b/web/regression/python_test_utils/test_utils.py
@@ -116,7 +116,7 @@ def clear_node_info_dict():
del node_info_dict[node][:]
-def create_database(server, db_name):
+def create_database(server, db_name, encoding=None):
"""This function used to create database and returns the database id"""
try:
connection = get_db_connection(
@@ -130,8 +130,14 @@ def create_database(server, db_name):
old_isolation_level = connection.isolation_level
connection.set_isolation_level(0)
pg_cursor = connection.cursor()
- pg_cursor.execute(
- '''CREATE DATABASE "%s" TEMPLATE template0''' % db_name)
+ if encoding is None:
+ pg_cursor.execute(
+ '''CREATE DATABASE "%s" TEMPLATE template0''' % db_name)
+ else:
+ pg_cursor.execute(
+ '''CREATE DATABASE "%s" TEMPLATE template0
+ ENCODING='%s' LC_COLLATE='%s' LC_CTYPE='%s' ''' %
+ (db_name, encoding[0], encoding[1], encoding[1]))
connection.set_isolation_level(old_isolation_level)
connection.commit()
^ permalink raw reply [nested|flat] 38+ messages in thread
* Re: [pgAdmin4][RM#3289] Can't query SQL_ASCII database.
@ 2018-06-04 09:54 Aditya Toshniwal <[email protected]>
parent: Aditya Toshniwal <[email protected]>
0 siblings, 1 reply; 38+ messages in thread
From: Aditya Toshniwal @ 2018-06-04 09:54 UTC (permalink / raw)
To: Dave Page <[email protected]>; +Cc: Joao De Almeida Pereira <[email protected]>; pgadmin-hackers
Hi Hackers,
Please ignore previous patch. Fixed some linter issues. PFA updated patch.
Thanks and Regards,
Aditya Toshniwal
Software Engineer | EnterpriseDB Software Solutions | Pune
"Don't Complain about Heat, Plant a tree"
On Mon, Jun 4, 2018 at 10:53 AM, Aditya Toshniwal <
[email protected]> wrote:
> Hi Hackers,
>
> PFA the updated patch on this. I have tried to add test cases to check for
> different encoding database. In the test run, it will create a database,
> fire a query for a string, check if we get the output and drops the
> database.
> Kindly review.
>
> Thanks and Regards,
> Aditya Toshniwal
> Software Engineer | EnterpriseDB Software Solutions | Pune
> "Don't Complain about Heat, Plant a tree"
>
> On Thu, May 31, 2018 at 6:42 PM, Dave Page <[email protected]> wrote:
>
>> Hi
>>
>> On Thu, May 31, 2018 at 1:20 AM, Aditya Toshniwal <
>> [email protected]> wrote:
>>
>>> Hi Victoria/Joao,
>>>
>>> On Thu, May 31, 2018 at 2:06 AM, Joao De Almeida Pereira <
>>> [email protected]> wrote:
>>>
>>>> Hello Aditya,
>>>>
>>>> It looks ok and it passes CI.
>>>>
>>>> We have some recommendations:
>>>> - These look like 2 different changes so they should be in separated
>>>> commits
>>>>
>>>
>>> If you are talking of set client_encoding, then its not a bug. Its a
>>> choice given to Postgres DB user to change the encoding of the characters.
>>> Postgres will translate characters from Server Encoding to Client Encoding,
>>> and will throw error like mentioned previously. This link will help better
>>> - https://www.postgresql.org/docs/10/static/multibyte.html
>>> The actual bug was, even after changing the client encoding to
>>> SQL_ASCII, pgAdmin4 was not able to show the output as it was failing in
>>> encoding by psycopg2. The patch is for resolving that.
>>>
>>>
>>>> - Do we have test coverage for the bug that you are talking about? If
>>>> not we should, to ensure this problem does not happen again in a future
>>>> change.
>>>>
>>>
>>> It is not possible adding test cases for encoding related stuff, as
>>> Postgres support a lot many different types of encoding and conversions
>>> (refer above link)
>>>
>>
>> I was going to ask the same thing. Per https://www.postgresql.org
>> /docs/10/static/multibyte.html#id-1.6.10.5.7, every characterset except
>> SQL_ASCII can be converted to UTF-8, so we only need to test that UTF-8 and
>> some other charactersets besides SQL_ASCII work, and then separately that
>> SQL_ASCII with characters known not to be in UTF-8 work.
>>
>>
>>>
>>>> Thanks
>>>> Victoria && Joao
>>>>
>>>> On Wed, May 30, 2018 at 3:06 AM Aditya Toshniwal <
>>>> [email protected]> wrote:
>>>>
>>>>> Hi Hackers,
>>>>>
>>>>> PFA updated patch after all the permutations, combinations for
>>>>> encoding for SQL_ASCII database. Also fixed a small glitch for sql
>>>>> editor connection status check.
>>>>>
>>>>> Please note, ERROR: invalid byte sequence for encoding "UTF8": 0xe9
>>>>> 0x73 is a Postgres DB error and not pgAdmin4 error.
>>>>>
>>>>> <Image Deleted>
>>>>>
>>>>> You need to change client_encoding to the appropriate. After changing
>>>>> client_encoding using command - set client_encoding='XYZ', it will give not
>>>>> give error.
>>>>>
>>>>> <Image Deleted>
>>>>>
>>>>> Thanks and Regards,
>>>>> Aditya Toshniwal
>>>>> Software Engineer | EnterpriseDB Software Solutions | Pune
>>>>> "Don't Complain about Heat, Plant a tree"
>>>>>
>>>>> On Wed, May 23, 2018 at 10:13 AM, Aditya Toshniwal <
>>>>> [email protected]> wrote:
>>>>>
>>>>>> Thank you Victoria, Anthony.
>>>>>>
>>>>>> Thanks and Regards,
>>>>>> Aditya Toshniwal
>>>>>> Software Engineer | EnterpriseDB Software Solutions | Pune
>>>>>> "Don't Complain about Heat, Plant a tree"
>>>>>>
>>>>>> On Tue, May 22, 2018 at 7:15 PM, Victoria Henry <[email protected]>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Aditya,
>>>>>>>
>>>>>>> We made a minor change to make the patch so the python linter can
>>>>>>> pass. Attached is the change we made.
>>>>>>> Everything else looks good.
>>>>>>>
>>>>>>> Sincerely,
>>>>>>>
>>>>>>> Victoria & Anthony
>>>>>>>
>>>>>>> On Tue, May 22, 2018 at 4:46 AM Aditya Toshniwal <
>>>>>>> [email protected]> wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> PFA updated patch. Linter issues are fixed ( we dont have any
>>>>>>>> linter setup for python :-( )
>>>>>>>> Regarding test cases, they run successfully on my system and the
>>>>>>>> reason it failed for pivotal is timeout exception. I am sorry I can't help
>>>>>>>> with that.
>>>>>>>>
>>>>>>>> Traceback (most recent call last):
>>>>>>>> File "/tmp/build/a453582b/pgadmin-repo/web/pgadmin/feature_tests/keyboard_shortcut_test.py",
>>>>>>>> line 52, in runTest
>>>>>>>> self._check_shortcuts()
>>>>>>>> File "/tmp/build/a453582b/pgadmin-repo/web/pgadmin/feature_tests/keyboard_shortcut_test.py",
>>>>>>>> line 77, in _check_shortcuts
>>>>>>>> ") and contains(@class, 'open')]")
>>>>>>>> File "/root/.pyenv/versions/pgadmin36/lib/python3.6/site-packages
>>>>>>>> /selenium/webdriver/support/wait.py", line 80, in until
>>>>>>>> raise TimeoutException(message, screen, stacktrace)
>>>>>>>> selenium.common.exceptions.TimeoutException: Message:
>>>>>>>>
>>>>>>>> Thanks and Regards,
>>>>>>>> Aditya Toshniwal
>>>>>>>> Software Engineer | EnterpriseDB Software Solutions | Pune
>>>>>>>> "Don't Complain about Heat, Plant a tree"
>>>>>>>>
>>>>>>>> On Tue, May 22, 2018 at 1:37 PM, Dave Page <[email protected]>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi
>>>>>>>>>
>>>>>>>>> Pivotal's buildbot is showing problems with this patch:
>>>>>>>>>
>>>>>>>>> https://gpdb-dev.bosh.pivotalci.info/teams/pgadmin/pipelines
>>>>>>>>> /pgadmin-patch/jobs/run-linter/builds/66 (linter failed)
>>>>>>>>> https://gpdb-dev.bosh.pivotalci.info/teams/pgadmin/pipelines
>>>>>>>>> /pgadmin-patch/jobs/run-tests/builds/84 (tests failed)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Tue, May 22, 2018 at 7:05 AM, Aditya Toshniwal <
>>>>>>>>> [email protected]> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Hackers,
>>>>>>>>>>
>>>>>>>>>> PFA patch for RM#3289 where decode error was thrown on querying a
>>>>>>>>>> SQL_ASCII database table. Please note, this problem occurs only on windows.
>>>>>>>>>> Sample insert - insert into test_tab values ('é');
>>>>>>>>>>
>>>>>>>>>> psycopg2 has a encodings dictionary where Postgres Database
>>>>>>>>>> Encodings are mapped to python equivalent. It uses 'ascii' decoder of
>>>>>>>>>> python to decode for SQL_ASCII encoding. If data has characters beyond the
>>>>>>>>>> limit of ascii then it failed. The solution would be to use utf_8 decoder
>>>>>>>>>> instead of ascii. I tried setting the client_encoding using
>>>>>>>>>> set_client_encoding('UTF8') method of a psycopg2 connection but no luck
>>>>>>>>>> (also its not allowed for async connection). I also tried executing "SET
>>>>>>>>>> CLIENT_ENCODING='UTF8'" but it didn't work too.
>>>>>>>>>> So, as in the patch, I had to set encodings dict value directly
>>>>>>>>>> to 'utf_8' and it seems to be working. Please note, the same is added to
>>>>>>>>>> psycopg3 milestones
>>>>>>>>>> https://github.com/psycopg/psycopg2/milestone/4
>>>>>>>>>>
>>>>>>>>>> Also fixed a small glitch for sql editor connection status check.
>>>>>>>>>>
>>>>>>>>>> Kindly review.
>>>>>>>>>>
>>>>>>>>>> Thanks and Regards,
>>>>>>>>>> Aditya Toshniwal
>>>>>>>>>> Software Engineer | EnterpriseDB Software Solutions | Pune
>>>>>>>>>> "Don't Complain about Heat, Plant a tree"
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Dave Page
>>>>>>>>> Blog: http://pgsnake.blogspot.com
>>>>>>>>> Twitter: @pgsnake
>>>>>>>>>
>>>>>>>>> EnterpriseDB UK: http://www.enterprisedb.com
>>>>>>>>> The Enterprise PostgreSQL Company
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>
>>>
>>
>>
>> --
>> Dave Page
>> Blog: http://pgsnake.blogspot.com
>> Twitter: @pgsnake
>>
>> EnterpriseDB UK: http://www.enterprisedb.com
>> The Enterprise PostgreSQL Company
>>
>
>
Attachments:
[text/x-patch] RM3289.patch (9.1K, 3-RM3289.patch)
download | inline diff:
diff --git a/web/pgadmin/tools/sqleditor/__init__.py b/web/pgadmin/tools/sqleditor/__init__.py
index 013e4dc..a9460dd 100644
--- a/web/pgadmin/tools/sqleditor/__init__.py
+++ b/web/pgadmin/tools/sqleditor/__init__.py
@@ -1480,19 +1480,24 @@ def query_tool_status(trans_id):
if conn and trans_obj and session_obj:
status = conn.transaction_status()
- # Check for the asynchronous notifies statements.
- conn.check_notifies(True)
- notifies = conn.get_notifies()
+ if status is not None:
+ # Check for the asynchronous notifies statements.
+ conn.check_notifies(True)
+ notifies = conn.get_notifies()
- return make_json_response(
- data={
- 'status': status,
- 'message': gettext(
- CONNECTION_STATUS_MESSAGE_MAPPING.get(status),
- ),
- 'notifies': notifies
- }
- )
+ return make_json_response(
+ data={
+ 'status': status,
+ 'message': gettext(
+ CONNECTION_STATUS_MESSAGE_MAPPING.get(status),
+ ),
+ 'notifies': notifies
+ }
+ )
+ else:
+ return internal_server_error(
+ errormsg=gettext("Transaction status check failed.")
+ )
else:
return internal_server_error(
errormsg=gettext("Transaction status check failed.")
diff --git a/web/pgadmin/tools/sqleditor/tests/test_encoding_charset.py b/web/pgadmin/tools/sqleditor/tests/test_encoding_charset.py
new file mode 100644
index 0000000..64155bf
--- /dev/null
+++ b/web/pgadmin/tools/sqleditor/tests/test_encoding_charset.py
@@ -0,0 +1,132 @@
+##########################################################################
+#
+# pgAdmin 4 - PostgreSQL Tools
+#
+# Copyright (C) 2013 - 2018, The pgAdmin Development Team
+# This software is released under the PostgreSQL Licence
+#
+##########################################################################
+
+import os
+import re
+
+from flask import Flask, render_template
+from jinja2 import FileSystemLoader
+
+from pgadmin import VersionedTemplateLoader
+from pgadmin.utils.route import BaseTestGenerator
+from pgadmin.utils.driver import get_driver
+from config import PG_DEFAULT_DRIVER
+from pgadmin.browser.server_groups.servers.databases.tests import utils as \
+ database_utils
+
+from regression import parent_node_dict
+from regression.python_test_utils import test_utils
+import json
+
+
+class TestEncodingCharset(BaseTestGenerator):
+ """
+ This class validates character support in pgAdmin4 for
+ different PostgresDB encodings
+ """
+ scenarios = [
+ (
+ 'With Encoding UTF8',
+ dict(
+ db_encoding='UTF8',
+ lc_collate='C',
+ test_str='A',
+ set_client_encoding=False
+ )),
+ (
+ 'With Encoding WIN1252',
+ dict(
+ db_encoding='WIN1252',
+ lc_collate='C',
+ test_str='A',
+ set_client_encoding=False
+ )),
+ (
+ 'With Encoding EUC_CN',
+ dict(
+ db_encoding='EUC_CN',
+ lc_collate='C',
+ test_str='A',
+ set_client_encoding=False
+ )),
+ (
+ 'With Encoding SQL_ASCII',
+ dict(
+ db_encoding='SQL_ASCII',
+ lc_collate='C',
+ test_str='\\255',
+ set_client_encoding=True
+ )),
+ ]
+
+ def setUp(self):
+ server_details = parent_node_dict['server'][-1]
+ server = server_details['server']
+ self.main_conn = test_utils.get_db_connection(
+ server['db'],
+ server['username'],
+ server['db_password'],
+ server['host'],
+ server['port'],
+ server['sslmode']
+ )
+ self.encode_db_name = 'encoding_test'
+ self.encode_sid = server_details['server_id']
+ self.encode_did = test_utils.create_database(
+ self.server, self.encode_db_name,
+ (self.db_encoding, self.lc_collate))
+
+ def runTest(self):
+
+ db_con = database_utils.connect_database(self,
+ test_utils.SERVER_GROUP,
+ self.encode_sid,
+ self.encode_did)
+ if not db_con["info"] == "Database connected.":
+ raise Exception("Could not connect to the database.")
+
+ # Initialize query tool
+ url = '/datagrid/initialize/query_tool/{0}/{1}/{2}'.format(
+ test_utils.SERVER_GROUP, self.encode_sid, self.encode_did)
+ response = self.tester.post(url)
+ self.assertEquals(response.status_code, 200)
+
+ response_data = json.loads(response.data.decode('utf-8'))
+ self.trans_id = response_data['data']['gridTransId']
+
+ if self.set_client_encoding:
+ # Change Client Encoding
+ url = "/sqleditor/query_tool/start/{0}".format(self.trans_id)
+ sql = "set client_encoding = '{0}';".format(self.db_encoding)
+ response = self.tester.post(url, data=json.dumps({"sql": sql}),
+ content_type='html/json')
+ self.assertEquals(response.status_code, 200)
+ url = '/sqleditor/poll/{0}'.format(self.trans_id)
+ response = self.tester.get(url)
+ self.assertEquals(response.status_code, 200)
+ response_data = json.loads(response.data.decode('utf-8'))
+
+ # Check character
+ url = "/sqleditor/query_tool/start/{0}".format(self.trans_id)
+ sql = "select E'{0}';".format(self.test_str)
+ response = self.tester.post(url, data=json.dumps({"sql": sql}),
+ content_type='html/json')
+ self.assertEquals(response.status_code, 200)
+ url = '/sqleditor/poll/{0}'.format(self.trans_id)
+ response = self.tester.get(url)
+ self.assertEquals(response.status_code, 200)
+ response_data = json.loads(response.data.decode('utf-8'))
+ self.assertEquals(response_data['data']['rows_fetched_to'], 1)
+
+ database_utils.disconnect_database(self, self.encode_sid,
+ self.encode_did)
+
+ def tearDown(self):
+ test_utils.drop_database(self.main_conn, self.encode_db_name)
+ self.main_conn.close()
diff --git a/web/pgadmin/utils/driver/psycopg2/connection.py b/web/pgadmin/utils/driver/psycopg2/connection.py
index cfd161a..e8ca886 100644
--- a/web/pgadmin/utils/driver/psycopg2/connection.py
+++ b/web/pgadmin/utils/driver/psycopg2/connection.py
@@ -51,6 +51,12 @@ else:
_ = gettext
+# Replace default ascii encoder with unicode-escape
+# which translates characters to unicode format.
+# Escape special characters to ASCII based on unicode
+encodings['SQL_ASCII'] = 'unicode-escape'
+encodings['SQLASCII'] = 'unicode-escape'
+
# Register global type caster which will be applicable to all connections.
register_global_typecasters()
diff --git a/web/pgadmin/utils/driver/psycopg2/typecast.py b/web/pgadmin/utils/driver/psycopg2/typecast.py
index f136604..ccebb2f 100644
--- a/web/pgadmin/utils/driver/psycopg2/typecast.py
+++ b/web/pgadmin/utils/driver/psycopg2/typecast.py
@@ -164,7 +164,7 @@ def register_global_typecasters():
def register_string_typecasters(connection):
- if connection.encoding != 'UTF8':
+ if connection.encoding not in ('UTF8', 'SQLASCII', 'SQL_ASCII'):
# In python3 when database encoding is other than utf-8 and client
# encoding is set to UNICODE then we need to map data from database
# encoding to utf-8.
diff --git a/web/regression/python_test_utils/test_utils.py b/web/regression/python_test_utils/test_utils.py
index 3e517b6..464a09e 100644
--- a/web/regression/python_test_utils/test_utils.py
+++ b/web/regression/python_test_utils/test_utils.py
@@ -116,7 +116,7 @@ def clear_node_info_dict():
del node_info_dict[node][:]
-def create_database(server, db_name):
+def create_database(server, db_name, encoding=None):
"""This function used to create database and returns the database id"""
try:
connection = get_db_connection(
@@ -130,8 +130,14 @@ def create_database(server, db_name):
old_isolation_level = connection.isolation_level
connection.set_isolation_level(0)
pg_cursor = connection.cursor()
- pg_cursor.execute(
- '''CREATE DATABASE "%s" TEMPLATE template0''' % db_name)
+ if encoding is None:
+ pg_cursor.execute(
+ '''CREATE DATABASE "%s" TEMPLATE template0''' % db_name)
+ else:
+ pg_cursor.execute(
+ '''CREATE DATABASE "%s" TEMPLATE template0
+ ENCODING='%s' LC_COLLATE='%s' LC_CTYPE='%s' ''' %
+ (db_name, encoding[0], encoding[1], encoding[1]))
connection.set_isolation_level(old_isolation_level)
connection.commit()
^ permalink raw reply [nested|flat] 38+ messages in thread
* Re: [pgAdmin4][RM#3289] Can't query SQL_ASCII database.
@ 2018-06-04 15:24 Joao De Almeida Pereira <[email protected]>
parent: Aditya Toshniwal <[email protected]>
0 siblings, 1 reply; 38+ messages in thread
From: Joao De Almeida Pereira @ 2018-06-04 15:24 UTC (permalink / raw)
To: Aditya Toshniwal <[email protected]>; +Cc: Dave Page <[email protected]>; pgadmin-hackers
Hi Aditya,
Looks like there are changes in this patch that are related to
notifications, these should go into a separate commit as they are not
related to encoding.
Also the tests are failing in our pipeline:
https://gpdb-dev.bosh.pivotalci.info/teams/pgadmin/pipelines/pgadmin-patch/jobs/run-tests/builds/110
Thanks
Victoria && Joao
On Mon, Jun 4, 2018 at 5:55 AM Aditya Toshniwal <
[email protected]> wrote:
> Hi Hackers,
>
> Please ignore previous patch. Fixed some linter issues. PFA updated patch.
>
> Thanks and Regards,
> Aditya Toshniwal
> Software Engineer | EnterpriseDB Software Solutions | Pune
> "Don't Complain about Heat, Plant a tree"
>
> On Mon, Jun 4, 2018 at 10:53 AM, Aditya Toshniwal <
> [email protected]> wrote:
>
>> Hi Hackers,
>>
>> PFA the updated patch on this. I have tried to add test cases to check
>> for different encoding database. In the test run, it will create a
>> database, fire a query for a string, check if we get the output and drops
>> the database.
>> Kindly review.
>>
>> Thanks and Regards,
>> Aditya Toshniwal
>> Software Engineer | EnterpriseDB Software Solutions | Pune
>> "Don't Complain about Heat, Plant a tree"
>>
>> On Thu, May 31, 2018 at 6:42 PM, Dave Page <[email protected]> wrote:
>>
>>> Hi
>>>
>>> On Thu, May 31, 2018 at 1:20 AM, Aditya Toshniwal <
>>> [email protected]> wrote:
>>>
>>>> Hi Victoria/Joao,
>>>>
>>>> On Thu, May 31, 2018 at 2:06 AM, Joao De Almeida Pereira <
>>>> [email protected]> wrote:
>>>>
>>>>> Hello Aditya,
>>>>>
>>>>> It looks ok and it passes CI.
>>>>>
>>>>> We have some recommendations:
>>>>> - These look like 2 different changes so they should be in separated
>>>>> commits
>>>>>
>>>>
>>>> If you are talking of set client_encoding, then its not a bug. Its a
>>>> choice given to Postgres DB user to change the encoding of the characters.
>>>> Postgres will translate characters from Server Encoding to Client Encoding,
>>>> and will throw error like mentioned previously. This link will help better
>>>> - https://www.postgresql.org/docs/10/static/multibyte.html
>>>> The actual bug was, even after changing the client encoding to
>>>> SQL_ASCII, pgAdmin4 was not able to show the output as it was failing in
>>>> encoding by psycopg2. The patch is for resolving that.
>>>>
>>>>
>>>>> - Do we have test coverage for the bug that you are talking about? If
>>>>> not we should, to ensure this problem does not happen again in a future
>>>>> change.
>>>>>
>>>>
>>>> It is not possible adding test cases for encoding related stuff, as
>>>> Postgres support a lot many different types of encoding and conversions
>>>> (refer above link)
>>>>
>>>
>>> I was going to ask the same thing. Per
>>> https://www.postgresql.org/docs/10/static/multibyte.html#id-1.6.10.5.7,
>>> every characterset except SQL_ASCII can be converted to UTF-8, so we only
>>> need to test that UTF-8 and some other charactersets besides SQL_ASCII
>>> work, and then separately that SQL_ASCII with characters known not to be in
>>> UTF-8 work.
>>>
>>>
>>>>
>>>>> Thanks
>>>>> Victoria && Joao
>>>>>
>>>>> On Wed, May 30, 2018 at 3:06 AM Aditya Toshniwal <
>>>>> [email protected]> wrote:
>>>>>
>>>>>> Hi Hackers,
>>>>>>
>>>>>> PFA updated patch after all the permutations, combinations for
>>>>>> encoding for SQL_ASCII database. Also fixed a small glitch for sql
>>>>>> editor connection status check.
>>>>>>
>>>>>> Please note, ERROR: invalid byte sequence for encoding "UTF8": 0xe9
>>>>>> 0x73 is a Postgres DB error and not pgAdmin4 error.
>>>>>>
>>>>>> <Image Deleted>
>>>>>>
>>>>>> You need to change client_encoding to the appropriate. After changing
>>>>>> client_encoding using command - set client_encoding='XYZ', it will give not
>>>>>> give error.
>>>>>>
>>>>>> <Image Deleted>
>>>>>>
>>>>>> Thanks and Regards,
>>>>>> Aditya Toshniwal
>>>>>> Software Engineer | EnterpriseDB Software Solutions | Pune
>>>>>> "Don't Complain about Heat, Plant a tree"
>>>>>>
>>>>>> On Wed, May 23, 2018 at 10:13 AM, Aditya Toshniwal <
>>>>>> [email protected]> wrote:
>>>>>>
>>>>>>> Thank you Victoria, Anthony.
>>>>>>>
>>>>>>> Thanks and Regards,
>>>>>>> Aditya Toshniwal
>>>>>>> Software Engineer | EnterpriseDB Software Solutions | Pune
>>>>>>> "Don't Complain about Heat, Plant a tree"
>>>>>>>
>>>>>>> On Tue, May 22, 2018 at 7:15 PM, Victoria Henry <[email protected]>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi Aditya,
>>>>>>>>
>>>>>>>> We made a minor change to make the patch so the python linter can
>>>>>>>> pass. Attached is the change we made.
>>>>>>>> Everything else looks good.
>>>>>>>>
>>>>>>>> Sincerely,
>>>>>>>>
>>>>>>>> Victoria & Anthony
>>>>>>>>
>>>>>>>> On Tue, May 22, 2018 at 4:46 AM Aditya Toshniwal <
>>>>>>>> [email protected]> wrote:
>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> PFA updated patch. Linter issues are fixed ( we dont have any
>>>>>>>>> linter setup for python :-( )
>>>>>>>>> Regarding test cases, they run successfully on my system and the
>>>>>>>>> reason it failed for pivotal is timeout exception. I am sorry I can't help
>>>>>>>>> with that.
>>>>>>>>>
>>>>>>>>> Traceback (most recent call last):
>>>>>>>>> File
>>>>>>>>> "/tmp/build/a453582b/pgadmin-repo/web/pgadmin/feature_tests/keyboard_shortcut_test.py",
>>>>>>>>> line 52, in runTest
>>>>>>>>> self._check_shortcuts()
>>>>>>>>> File
>>>>>>>>> "/tmp/build/a453582b/pgadmin-repo/web/pgadmin/feature_tests/keyboard_shortcut_test.py",
>>>>>>>>> line 77, in _check_shortcuts
>>>>>>>>> ") and contains(@class, 'open')]")
>>>>>>>>> File
>>>>>>>>> "/root/.pyenv/versions/pgadmin36/lib/python3.6/site-packages/selenium/webdriver/support/wait.py",
>>>>>>>>> line 80, in until
>>>>>>>>> raise TimeoutException(message, screen, stacktrace)
>>>>>>>>> selenium.common.exceptions.TimeoutException: Message:
>>>>>>>>>
>>>>>>>>> Thanks and Regards,
>>>>>>>>> Aditya Toshniwal
>>>>>>>>> Software Engineer | EnterpriseDB Software Solutions | Pune
>>>>>>>>> "Don't Complain about Heat, Plant a tree"
>>>>>>>>>
>>>>>>>>> On Tue, May 22, 2018 at 1:37 PM, Dave Page <[email protected]>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hi
>>>>>>>>>>
>>>>>>>>>> Pivotal's buildbot is showing problems with this patch:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> https://gpdb-dev.bosh.pivotalci.info/teams/pgadmin/pipelines/pgadmin-patch/jobs/run-linter/builds/66
>>>>>>>>>> (linter failed)
>>>>>>>>>>
>>>>>>>>>> https://gpdb-dev.bosh.pivotalci.info/teams/pgadmin/pipelines/pgadmin-patch/jobs/run-tests/builds/84
>>>>>>>>>> (tests failed)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Tue, May 22, 2018 at 7:05 AM, Aditya Toshniwal <
>>>>>>>>>> [email protected]> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Hackers,
>>>>>>>>>>>
>>>>>>>>>>> PFA patch for RM#3289 where decode error was thrown on querying
>>>>>>>>>>> a SQL_ASCII database table. Please note, this problem occurs only on
>>>>>>>>>>> windows.
>>>>>>>>>>> Sample insert - insert into test_tab values ('é');
>>>>>>>>>>>
>>>>>>>>>>> psycopg2 has a encodings dictionary where Postgres Database
>>>>>>>>>>> Encodings are mapped to python equivalent. It uses 'ascii' decoder of
>>>>>>>>>>> python to decode for SQL_ASCII encoding. If data has characters beyond the
>>>>>>>>>>> limit of ascii then it failed. The solution would be to use utf_8 decoder
>>>>>>>>>>> instead of ascii. I tried setting the client_encoding using
>>>>>>>>>>> set_client_encoding('UTF8') method of a psycopg2 connection but no luck
>>>>>>>>>>> (also its not allowed for async connection). I also tried executing "SET
>>>>>>>>>>> CLIENT_ENCODING='UTF8'" but it didn't work too.
>>>>>>>>>>> So, as in the patch, I had to set encodings dict value directly
>>>>>>>>>>> to 'utf_8' and it seems to be working. Please note, the same is added to
>>>>>>>>>>> psycopg3 milestones
>>>>>>>>>>> https://github.com/psycopg/psycopg2/milestone/4
>>>>>>>>>>>
>>>>>>>>>>> Also fixed a small glitch for sql editor connection status check.
>>>>>>>>>>>
>>>>>>>>>>> Kindly review.
>>>>>>>>>>>
>>>>>>>>>>> Thanks and Regards,
>>>>>>>>>>> Aditya Toshniwal
>>>>>>>>>>> Software Engineer | EnterpriseDB Software Solutions | Pune
>>>>>>>>>>> "Don't Complain about Heat, Plant a tree"
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Dave Page
>>>>>>>>>> Blog: http://pgsnake.blogspot.com
>>>>>>>>>> Twitter: @pgsnake
>>>>>>>>>>
>>>>>>>>>> EnterpriseDB UK: http://www.enterprisedb.com
>>>>>>>>>> The Enterprise PostgreSQL Company
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>
>>>>
>>>
>>>
>>> --
>>> Dave Page
>>> Blog: http://pgsnake.blogspot.com
>>> Twitter: @pgsnake
>>>
>>> EnterpriseDB UK: http://www.enterprisedb.com
>>> The Enterprise PostgreSQL Company
>>>
>>
>>
>
^ permalink raw reply [nested|flat] 38+ messages in thread
* Re: [pgAdmin4][RM#3289] Can't query SQL_ASCII database.
@ 2018-06-04 16:44 Aditya Toshniwal <[email protected]>
parent: Joao De Almeida Pereira <[email protected]>
0 siblings, 1 reply; 38+ messages in thread
From: Aditya Toshniwal @ 2018-06-04 16:44 UTC (permalink / raw)
To: Joao De Almeida Pereira <[email protected]>; +Cc: Dave Page <[email protected]>; pgadmin-hackers
Hi Victoria/Joao,
There is no change related to notifications in this patch.
The below code is minor fix related to connection status of sql editor. Can
you please share the code snippet if it is not the below.
- # Check for the asynchronous notifies statements.
- conn.check_notifies(True)
- notifies = conn.get_notifies()
+ if status is not None:
+ # Check for the asynchronous notifies statements.
+ conn.check_notifies(True)
+ notifies = conn.get_notifies()
The test cases work perfectly fine on my machine. I will try if I can
simulate the failure on some other machines.
Thanks and Regards,
Aditya Toshniwal
Software Engineer | EnterpriseDB Software Solutions | Pune
"Don't Complain about Heat, Plant a tree"
On Mon, Jun 4, 2018 at 8:54 PM, Joao De Almeida Pereira <
[email protected]> wrote:
> Hi Aditya,
> Looks like there are changes in this patch that are related to
> notifications, these should go into a separate commit as they are not
> related to encoding.
> Also the tests are failing in our pipeline: https://gpdb-dev.
> bosh.pivotalci.info/teams/pgadmin/pipelines/pgadmin-
> patch/jobs/run-tests/builds/110
>
> Thanks
> Victoria && Joao
>
> On Mon, Jun 4, 2018 at 5:55 AM Aditya Toshniwal <aditya.toshniwal@
> enterprisedb.com> wrote:
>
>> Hi Hackers,
>>
>> Please ignore previous patch. Fixed some linter issues. PFA updated patch.
>>
>> Thanks and Regards,
>> Aditya Toshniwal
>> Software Engineer | EnterpriseDB Software Solutions | Pune
>> "Don't Complain about Heat, Plant a tree"
>>
>> On Mon, Jun 4, 2018 at 10:53 AM, Aditya Toshniwal <aditya.toshniwal@
>> enterprisedb.com> wrote:
>>
>>> Hi Hackers,
>>>
>>> PFA the updated patch on this. I have tried to add test cases to check
>>> for different encoding database. In the test run, it will create a
>>> database, fire a query for a string, check if we get the output and drops
>>> the database.
>>> Kindly review.
>>>
>>> Thanks and Regards,
>>> Aditya Toshniwal
>>> Software Engineer | EnterpriseDB Software Solutions | Pune
>>> "Don't Complain about Heat, Plant a tree"
>>>
>>> On Thu, May 31, 2018 at 6:42 PM, Dave Page <[email protected]> wrote:
>>>
>>>> Hi
>>>>
>>>> On Thu, May 31, 2018 at 1:20 AM, Aditya Toshniwal <aditya.toshniwal@
>>>> enterprisedb.com> wrote:
>>>>
>>>>> Hi Victoria/Joao,
>>>>>
>>>>> On Thu, May 31, 2018 at 2:06 AM, Joao De Almeida Pereira <
>>>>> [email protected]> wrote:
>>>>>
>>>>>> Hello Aditya,
>>>>>>
>>>>>> It looks ok and it passes CI.
>>>>>>
>>>>>> We have some recommendations:
>>>>>> - These look like 2 different changes so they should be in separated
>>>>>> commits
>>>>>>
>>>>>
>>>>> If you are talking of set client_encoding, then its not a bug. Its a
>>>>> choice given to Postgres DB user to change the encoding of the characters.
>>>>> Postgres will translate characters from Server Encoding to Client Encoding,
>>>>> and will throw error like mentioned previously. This link will help better
>>>>> - https://www.postgresql.org/docs/10/static/multibyte.html
>>>>> The actual bug was, even after changing the client encoding to
>>>>> SQL_ASCII, pgAdmin4 was not able to show the output as it was failing in
>>>>> encoding by psycopg2. The patch is for resolving that.
>>>>>
>>>>>
>>>>>> - Do we have test coverage for the bug that you are talking about? If
>>>>>> not we should, to ensure this problem does not happen again in a future
>>>>>> change.
>>>>>>
>>>>>
>>>>> It is not possible adding test cases for encoding related stuff, as
>>>>> Postgres support a lot many different types of encoding and conversions
>>>>> (refer above link)
>>>>>
>>>>
>>>> I was going to ask the same thing. Per https://www.postgresql.
>>>> org/docs/10/static/multibyte.html#id-1.6.10.5.7, every characterset
>>>> except SQL_ASCII can be converted to UTF-8, so we only need to test that
>>>> UTF-8 and some other charactersets besides SQL_ASCII work, and then
>>>> separately that SQL_ASCII with characters known not to be in UTF-8 work.
>>>>
>>>>
>>>>>
>>>>>> Thanks
>>>>>> Victoria && Joao
>>>>>>
>>>>>> On Wed, May 30, 2018 at 3:06 AM Aditya Toshniwal <aditya.toshniwal@
>>>>>> enterprisedb.com> wrote:
>>>>>>
>>>>>>> Hi Hackers,
>>>>>>>
>>>>>>> PFA updated patch after all the permutations, combinations for
>>>>>>> encoding for SQL_ASCII database. Also fixed a small glitch for sql
>>>>>>> editor connection status check.
>>>>>>>
>>>>>>> Please note, ERROR: invalid byte sequence for encoding "UTF8": 0xe9
>>>>>>> 0x73 is a Postgres DB error and not pgAdmin4 error.
>>>>>>>
>>>>>>> <Image Deleted>
>>>>>>>
>>>>>>> You need to change client_encoding to the appropriate. After
>>>>>>> changing client_encoding using command - set client_encoding='XYZ', it will
>>>>>>> give not give error.
>>>>>>>
>>>>>>> <Image Deleted>
>>>>>>>
>>>>>>> Thanks and Regards,
>>>>>>> Aditya Toshniwal
>>>>>>> Software Engineer | EnterpriseDB Software Solutions | Pune
>>>>>>> "Don't Complain about Heat, Plant a tree"
>>>>>>>
>>>>>>> On Wed, May 23, 2018 at 10:13 AM, Aditya Toshniwal <
>>>>>>> [email protected]> wrote:
>>>>>>>
>>>>>>>> Thank you Victoria, Anthony.
>>>>>>>>
>>>>>>>> Thanks and Regards,
>>>>>>>> Aditya Toshniwal
>>>>>>>> Software Engineer | EnterpriseDB Software Solutions | Pune
>>>>>>>> "Don't Complain about Heat, Plant a tree"
>>>>>>>>
>>>>>>>> On Tue, May 22, 2018 at 7:15 PM, Victoria Henry <[email protected]>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi Aditya,
>>>>>>>>>
>>>>>>>>> We made a minor change to make the patch so the python linter can
>>>>>>>>> pass. Attached is the change we made.
>>>>>>>>> Everything else looks good.
>>>>>>>>>
>>>>>>>>> Sincerely,
>>>>>>>>>
>>>>>>>>> Victoria & Anthony
>>>>>>>>>
>>>>>>>>> On Tue, May 22, 2018 at 4:46 AM Aditya Toshniwal <
>>>>>>>>> [email protected]> wrote:
>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> PFA updated patch. Linter issues are fixed ( we dont have any
>>>>>>>>>> linter setup for python :-( )
>>>>>>>>>> Regarding test cases, they run successfully on my system and the
>>>>>>>>>> reason it failed for pivotal is timeout exception. I am sorry I can't help
>>>>>>>>>> with that.
>>>>>>>>>>
>>>>>>>>>> Traceback (most recent call last):
>>>>>>>>>> File "/tmp/build/a453582b/pgadmin-repo/web/pgadmin/feature_
>>>>>>>>>> tests/keyboard_shortcut_test.py", line 52, in runTest
>>>>>>>>>> self._check_shortcuts()
>>>>>>>>>> File "/tmp/build/a453582b/pgadmin-repo/web/pgadmin/feature_
>>>>>>>>>> tests/keyboard_shortcut_test.py", line 77, in _check_shortcuts
>>>>>>>>>> ") and contains(@class, 'open')]")
>>>>>>>>>> File "/root/.pyenv/versions/pgadmin36/lib/python3.6/site-
>>>>>>>>>> packages/selenium/webdriver/support/wait.py", line 80, in until
>>>>>>>>>> raise TimeoutException(message, screen, stacktrace)
>>>>>>>>>> selenium.common.exceptions.TimeoutException: Message:
>>>>>>>>>>
>>>>>>>>>> Thanks and Regards,
>>>>>>>>>> Aditya Toshniwal
>>>>>>>>>> Software Engineer | EnterpriseDB Software Solutions | Pune
>>>>>>>>>> "Don't Complain about Heat, Plant a tree"
>>>>>>>>>>
>>>>>>>>>> On Tue, May 22, 2018 at 1:37 PM, Dave Page <[email protected]>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi
>>>>>>>>>>>
>>>>>>>>>>> Pivotal's buildbot is showing problems with this patch:
>>>>>>>>>>>
>>>>>>>>>>> https://gpdb-dev.bosh.pivotalci.info/teams/pgadmin/
>>>>>>>>>>> pipelines/pgadmin-patch/jobs/run-linter/builds/66 (linter
>>>>>>>>>>> failed)
>>>>>>>>>>> https://gpdb-dev.bosh.pivotalci.info/teams/pgadmin/
>>>>>>>>>>> pipelines/pgadmin-patch/jobs/run-tests/builds/84 (tests failed)
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Tue, May 22, 2018 at 7:05 AM, Aditya Toshniwal <
>>>>>>>>>>> [email protected]> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi Hackers,
>>>>>>>>>>>>
>>>>>>>>>>>> PFA patch for RM#3289 where decode error was thrown on querying
>>>>>>>>>>>> a SQL_ASCII database table. Please note, this problem occurs only on
>>>>>>>>>>>> windows.
>>>>>>>>>>>> Sample insert - insert into test_tab values ('é');
>>>>>>>>>>>>
>>>>>>>>>>>> psycopg2 has a encodings dictionary where Postgres Database
>>>>>>>>>>>> Encodings are mapped to python equivalent. It uses 'ascii' decoder of
>>>>>>>>>>>> python to decode for SQL_ASCII encoding. If data has characters beyond the
>>>>>>>>>>>> limit of ascii then it failed. The solution would be to use utf_8 decoder
>>>>>>>>>>>> instead of ascii. I tried setting the client_encoding using
>>>>>>>>>>>> set_client_encoding('UTF8') method of a psycopg2 connection but no luck
>>>>>>>>>>>> (also its not allowed for async connection). I also tried executing "SET
>>>>>>>>>>>> CLIENT_ENCODING='UTF8'" but it didn't work too.
>>>>>>>>>>>> So, as in the patch, I had to set encodings dict value directly
>>>>>>>>>>>> to 'utf_8' and it seems to be working. Please note, the same is added to
>>>>>>>>>>>> psycopg3 milestones
>>>>>>>>>>>> https://github.com/psycopg/psycopg2/milestone/4
>>>>>>>>>>>>
>>>>>>>>>>>> Also fixed a small glitch for sql editor connection status
>>>>>>>>>>>> check.
>>>>>>>>>>>>
>>>>>>>>>>>> Kindly review.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks and Regards,
>>>>>>>>>>>> Aditya Toshniwal
>>>>>>>>>>>> Software Engineer | EnterpriseDB Software Solutions | Pune
>>>>>>>>>>>> "Don't Complain about Heat, Plant a tree"
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Dave Page
>>>>>>>>>>> Blog: http://pgsnake.blogspot.com
>>>>>>>>>>> Twitter: @pgsnake
>>>>>>>>>>>
>>>>>>>>>>> EnterpriseDB UK: http://www.enterprisedb.com
>>>>>>>>>>> The Enterprise PostgreSQL Company
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Dave Page
>>>> Blog: http://pgsnake.blogspot.com
>>>> Twitter: @pgsnake
>>>>
>>>> EnterpriseDB UK: http://www.enterprisedb.com
>>>> The Enterprise PostgreSQL Company
>>>>
>>>
>>>
>>
^ permalink raw reply [nested|flat] 38+ messages in thread
* Re: [pgAdmin4][RM#3289] Can't query SQL_ASCII database.
@ 2018-06-04 19:38 Joao De Almeida Pereira <[email protected]>
parent: Aditya Toshniwal <[email protected]>
0 siblings, 1 reply; 38+ messages in thread
From: Joao De Almeida Pereira @ 2018-06-04 19:38 UTC (permalink / raw)
To: Aditya Toshniwal <[email protected]>; +Cc: Dave Page <[email protected]>; pgadmin-hackers
Hello Aditya,
>
>
> There is no change related to notifications in this patch.
> The below code is minor fix related to connection status of sql editor.
> Can you please share the code snippet if it is not the below.
>
> - # Check for the asynchronous notifies statements.
> - conn.check_notifies(True)
> - notifies = conn.get_notifies()
> + if status is not None:
> + # Check for the asynchronous notifies statements.
> + conn.check_notifies(True)
> + notifies = conn.get_notifies()
>
>
This is a minor fix, but is it related to querying SQL_ASCII database?
Thanks
Victoria && Joao
^ permalink raw reply [nested|flat] 38+ messages in thread
* Re: [pgAdmin4][RM#3289] Can't query SQL_ASCII database.
@ 2018-06-05 04:45 Aditya Toshniwal <[email protected]>
parent: Joao De Almeida Pereira <[email protected]>
0 siblings, 1 reply; 38+ messages in thread
From: Aditya Toshniwal @ 2018-06-05 04:45 UTC (permalink / raw)
To: Joao De Almeida Pereira <[email protected]>; +Cc: Dave Page <[email protected]>; pgadmin-hackers
Hi
On Tue, Jun 5, 2018 at 1:08 AM, Joao De Almeida Pereira <
[email protected]> wrote:
> Hello Aditya,
>
>>
>>
>> There is no change related to notifications in this patch.
>> The below code is minor fix related to connection status of sql editor.
>> Can you please share the code snippet if it is not the below.
>>
>> - # Check for the asynchronous notifies statements.
>> - conn.check_notifies(True)
>> - notifies = conn.get_notifies()
>> + if status is not None:
>> + # Check for the asynchronous notifies statements.
>> + conn.check_notifies(True)
>> + notifies = conn.get_notifies()
>>
>>
> This is a minor fix, but is it related to querying SQL_ASCII database?
>
No its not. It is something I found when I was working on SQL_ASCII related
changes.
Well then, will send a separate patch for it.
>
> Thanks
> Victoria && Joao
>
^ permalink raw reply [nested|flat] 38+ messages in thread
* Re: [pgAdmin4][RM#3289] Can't query SQL_ASCII database.
@ 2018-06-05 08:50 Aditya Toshniwal <[email protected]>
parent: Aditya Toshniwal <[email protected]>
0 siblings, 1 reply; 38+ messages in thread
From: Aditya Toshniwal @ 2018-06-05 08:50 UTC (permalink / raw)
To: Joao De Almeida Pereira <[email protected]>; +Cc: Dave Page <[email protected]>; pgadmin-hackers
Hi Hackers,
PFA updated patch. The sqleditor change is sent separately and removed from
current patch as suggested.
The test cases were running fine when the module was specified using --pkg
but were failing in complete run. Fixed that.
Kindly review.
Thanks and Regards,
Aditya Toshniwal
Software Engineer | EnterpriseDB Software Solutions | Pune
"Don't Complain about Heat, Plant a tree"
On Tue, Jun 5, 2018 at 10:15 AM, Aditya Toshniwal <
[email protected]> wrote:
> Hi
>
> On Tue, Jun 5, 2018 at 1:08 AM, Joao De Almeida Pereira <
> [email protected]> wrote:
>
>> Hello Aditya,
>>
>>>
>>>
>>> There is no change related to notifications in this patch.
>>> The below code is minor fix related to connection status of sql editor.
>>> Can you please share the code snippet if it is not the below.
>>>
>>> - # Check for the asynchronous notifies statements.
>>> - conn.check_notifies(True)
>>> - notifies = conn.get_notifies()
>>> + if status is not None:
>>> + # Check for the asynchronous notifies statements.
>>> + conn.check_notifies(True)
>>> + notifies = conn.get_notifies()
>>>
>>>
>> This is a minor fix, but is it related to querying SQL_ASCII database?
>>
> No its not. It is something I found when I was working on SQL_ASCII
> related changes.
> Well then, will send a separate patch for it.
>
>>
>> Thanks
>> Victoria && Joao
>>
>
>
Attachments:
[application/octet-stream] RM3289.patch (7.9K, 3-RM3289.patch)
download | inline diff:
diff --git a/web/pgadmin/tools/sqleditor/tests/test_encoding_charset.py b/web/pgadmin/tools/sqleditor/tests/test_encoding_charset.py
new file mode 100644
index 00000000..833e471e
--- /dev/null
+++ b/web/pgadmin/tools/sqleditor/tests/test_encoding_charset.py
@@ -0,0 +1,121 @@
+##########################################################################
+#
+# pgAdmin 4 - PostgreSQL Tools
+#
+# Copyright (C) 2013 - 2018, The pgAdmin Development Team
+# This software is released under the PostgreSQL Licence
+#
+##########################################################################
+
+from pgadmin.utils.route import BaseTestGenerator
+from pgadmin.browser.server_groups.servers.databases.tests import utils as \
+ database_utils
+from regression import parent_node_dict
+from regression.python_test_utils import test_utils
+import json
+
+
+class TestEncodingCharset(BaseTestGenerator):
+ """
+ This class validates character support in pgAdmin4 for
+ different PostgresDB encodings
+ """
+ scenarios = [
+ (
+ 'With Encoding UTF8',
+ dict(
+ db_encoding='UTF8',
+ lc_collate='C',
+ test_str='A',
+ set_client_encoding=False
+ )),
+ (
+ 'With Encoding WIN1252',
+ dict(
+ db_encoding='WIN1252',
+ lc_collate='C',
+ test_str='A',
+ set_client_encoding=False
+ )),
+ (
+ 'With Encoding EUC_CN',
+ dict(
+ db_encoding='EUC_CN',
+ lc_collate='C',
+ test_str='A',
+ set_client_encoding=False
+ )),
+ (
+ 'With Encoding SQL_ASCII',
+ dict(
+ db_encoding='SQL_ASCII',
+ lc_collate='C',
+ test_str='\\255',
+ set_client_encoding=True
+ )),
+ ]
+
+ def setUp(self):
+ self.encode_db_name = 'encoding_' + self.db_encoding
+ self.encode_sid = self.server_information['server_id']
+ self.encode_did = test_utils.create_database(
+ self.server, self.encode_db_name,
+ (self.db_encoding, self.lc_collate))
+
+ def runTest(self):
+
+ db_con = database_utils.connect_database(self,
+ test_utils.SERVER_GROUP,
+ self.encode_sid,
+ self.encode_did)
+ if not db_con["info"] == "Database connected.":
+ raise Exception("Could not connect to the database.")
+
+ # Initialize query tool
+ url = '/datagrid/initialize/query_tool/{0}/{1}/{2}'.format(
+ test_utils.SERVER_GROUP, self.encode_sid, self.encode_did)
+ response = self.tester.post(url)
+ self.assertEquals(response.status_code, 200)
+
+ response_data = json.loads(response.data.decode('utf-8'))
+ self.trans_id = response_data['data']['gridTransId']
+
+ if self.set_client_encoding:
+ # Change Client Encoding
+ url = "/sqleditor/query_tool/start/{0}".format(self.trans_id)
+ sql = "set client_encoding = '{0}';".format(self.db_encoding)
+ response = self.tester.post(url, data=json.dumps({"sql": sql}),
+ content_type='html/json')
+ self.assertEquals(response.status_code, 200)
+ url = '/sqleditor/poll/{0}'.format(self.trans_id)
+ response = self.tester.get(url)
+ self.assertEquals(response.status_code, 200)
+ response_data = json.loads(response.data.decode('utf-8'))
+
+ # Check character
+ url = "/sqleditor/query_tool/start/{0}".format(self.trans_id)
+ sql = "select E'{0}';".format(self.test_str)
+ response = self.tester.post(url, data=json.dumps({"sql": sql}),
+ content_type='html/json')
+ self.assertEquals(response.status_code, 200)
+ url = '/sqleditor/poll/{0}'.format(self.trans_id)
+ response = self.tester.get(url)
+ self.assertEquals(response.status_code, 200)
+ response_data = json.loads(response.data.decode('utf-8'))
+ self.assertEquals(response_data['data']['rows_fetched_to'], 1)
+
+ database_utils.disconnect_database(self, self.encode_sid,
+ self.encode_did)
+
+ def tearDown(self):
+ server = [server['server'] for server in parent_node_dict['server']
+ if server['server_id'] == self.encode_sid][0]
+ main_conn = test_utils.get_db_connection(
+ server['db'],
+ server['username'],
+ server['db_password'],
+ server['host'],
+ server['port'],
+ server['sslmode']
+ )
+ test_utils.drop_database(main_conn, self.encode_db_name)
diff --git a/web/pgadmin/utils/driver/psycopg2/connection.py b/web/pgadmin/utils/driver/psycopg2/connection.py
index cfd161a0..e8ca886e 100644
--- a/web/pgadmin/utils/driver/psycopg2/connection.py
+++ b/web/pgadmin/utils/driver/psycopg2/connection.py
@@ -51,6 +51,12 @@ else:
_ = gettext
+# Replace default ascii encoder with unicode-escape
+# which translates characters to unicode format.
+# Escape special characters to ASCII based on unicode
+encodings['SQL_ASCII'] = 'unicode-escape'
+encodings['SQLASCII'] = 'unicode-escape'
+
# Register global type caster which will be applicable to all connections.
register_global_typecasters()
diff --git a/web/pgadmin/utils/driver/psycopg2/typecast.py b/web/pgadmin/utils/driver/psycopg2/typecast.py
index f1366049..a3fa2bc5 100644
--- a/web/pgadmin/utils/driver/psycopg2/typecast.py
+++ b/web/pgadmin/utils/driver/psycopg2/typecast.py
@@ -164,7 +164,7 @@ def register_global_typecasters():
def register_string_typecasters(connection):
- if connection.encoding != 'UTF8':
+ if connection.encoding not in ('UTF8', 'SQLASCII', 'SQL_ASCII'):
# In python3 when database encoding is other than utf-8 and client
# encoding is set to UNICODE then we need to map data from database
# encoding to utf-8.
@@ -202,8 +202,8 @@ def register_string_typecasters(connection):
(1002, 1003, 1009, 1014, 1015, 0
), 'UNICODEARRAY', unicode_type)
- psycopg2.extensions.register_type(unicode_type)
- psycopg2.extensions.register_type(unicode_array_type)
+ psycopg2.extensions.register_type(unicode_type, connection)
+ psycopg2.extensions.register_type(unicode_array_type, connection)
def register_binary_typecasters(connection):
diff --git a/web/regression/python_test_utils/test_utils.py b/web/regression/python_test_utils/test_utils.py
index 3e517b61..464a09e1 100644
--- a/web/regression/python_test_utils/test_utils.py
+++ b/web/regression/python_test_utils/test_utils.py
@@ -116,7 +116,7 @@ def clear_node_info_dict():
del node_info_dict[node][:]
-def create_database(server, db_name):
+def create_database(server, db_name, encoding=None):
"""This function used to create database and returns the database id"""
try:
connection = get_db_connection(
@@ -130,8 +130,14 @@ def create_database(server, db_name):
old_isolation_level = connection.isolation_level
connection.set_isolation_level(0)
pg_cursor = connection.cursor()
- pg_cursor.execute(
- '''CREATE DATABASE "%s" TEMPLATE template0''' % db_name)
+ if encoding is None:
+ pg_cursor.execute(
+ '''CREATE DATABASE "%s" TEMPLATE template0''' % db_name)
+ else:
+ pg_cursor.execute(
+ '''CREATE DATABASE "%s" TEMPLATE template0
+ ENCODING='%s' LC_COLLATE='%s' LC_CTYPE='%s' ''' %
+ (db_name, encoding[0], encoding[1], encoding[1]))
connection.set_isolation_level(old_isolation_level)
connection.commit()
^ permalink raw reply [nested|flat] 38+ messages in thread
* Re: [pgAdmin4][RM#3289] Can't query SQL_ASCII database.
@ 2018-06-05 11:26 Dave Page <[email protected]>
parent: Aditya Toshniwal <[email protected]>
0 siblings, 1 reply; 38+ messages in thread
From: Dave Page @ 2018-06-05 11:26 UTC (permalink / raw)
To: Aditya Toshniwal <[email protected]>; +Cc: Joao De Almeida Pereira <[email protected]>; pgadmin-hackers
Hi
On Tue, Jun 5, 2018 at 9:50 AM, Aditya Toshniwal <
[email protected]> wrote:
> Hi Hackers,
>
> PFA updated patch. The sqleditor change is sent separately and removed
> from current patch as suggested.
> The test cases were running fine when the module was specified using --pkg
> but were failing in complete run. Fixed that.
>
I did a quick test by creating a SQL_ASCII database containing a simple
table:
CREATE TABLE sql_ascii (id serial primary key, data text);
And then populated it with data:
/Library/PostgreSQL/9.4/bin/psql -d sql_ascii -U postgres -c "INSERT INTO
sql_acsii (data) VALUES ('[Windows-1252] Euro: \x80 Double dagger:
\x87');"
/Library/PostgreSQL/9.4/bin/psql -d sql_ascii -U postgres -c "INSERT INTO
sql_ascii (data) VALUES ('[Latin-1] Yen: \xa5 Half: \xbd');"
/Library/PostgreSQL/9.4/bin/psql -d sql_ascii -U postgres -c "INSERT INTO
sql_ascii (data) VALUES ('[Japanese] Ship: \xe8\x88\xb9');"
/Library/PostgreSQL/9.4/bin/psql -d sql_ascii -U postgres -c "INSERT INTO
sql_ascii (data) VALUES ('[Invalid UTF-8] Blob: \xf4\xa5\xa3\xa5');"
I then right-clicked the table in the treeview, and selected the option to
view all rows, and immediately saw an error:
2018-06-05 12:23:27,319: SQL pgadmin: Execute (async) for server #1 -
CONN:1187535 (Query-id: 8522474):
SELECT * FROM public.sql_ascii
ORDER BY id ASC
2018-06-05 12:23:27,320: ERROR pgadmin: Failed to execute query
(execute_async) for the server #1 - CONN:1187535(Query-id: 8522474):
Error Message:ERROR: invalid byte sequence for encoding "UTF8": 0x80
SQL state: 22021
Running "SELECT * FROM sql_ascii" in the query tool resulted in the same
error, however, if I ran "SET client_encoding = 'SQL_ASCII';" first, I do
see results.
I have confirmed that I've restarted the server after applying the patch.
What am I missing? Why don't we just set the client_encoding to SQL_ASCII
if it's a SQL_ASCII database?
Note that this testing was on Python 2.7.10 on MacOS.
>
> Kindly review.
>
> Thanks and Regards,
> Aditya Toshniwal
> Software Engineer | EnterpriseDB Software Solutions | Pune
> "Don't Complain about Heat, Plant a tree"
>
> On Tue, Jun 5, 2018 at 10:15 AM, Aditya Toshniwal <aditya.toshniwal@
> enterprisedb.com> wrote:
>
>> Hi
>>
>> On Tue, Jun 5, 2018 at 1:08 AM, Joao De Almeida Pereira <
>> [email protected]> wrote:
>>
>>> Hello Aditya,
>>>
>>>>
>>>>
>>>> There is no change related to notifications in this patch.
>>>> The below code is minor fix related to connection status of sql editor.
>>>> Can you please share the code snippet if it is not the below.
>>>>
>>>> - # Check for the asynchronous notifies statements.
>>>> - conn.check_notifies(True)
>>>> - notifies = conn.get_notifies()
>>>> + if status is not None:
>>>> + # Check for the asynchronous notifies statements.
>>>> + conn.check_notifies(True)
>>>> + notifies = conn.get_notifies()
>>>>
>>>>
>>> This is a minor fix, but is it related to querying SQL_ASCII database?
>>>
>> No its not. It is something I found when I was working on SQL_ASCII
>> related changes.
>> Well then, will send a separate patch for it.
>>
>>>
>>> Thanks
>>> Victoria && Joao
>>>
>>
>>
>
--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake
EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 38+ messages in thread
* Re: [pgAdmin4][RM#3289] Can't query SQL_ASCII database.
@ 2018-06-05 12:21 Aditya Toshniwal <[email protected]>
parent: Dave Page <[email protected]>
0 siblings, 1 reply; 38+ messages in thread
From: Aditya Toshniwal @ 2018-06-05 12:21 UTC (permalink / raw)
To: Dave Page <[email protected]>; +Cc: Joao De Almeida Pereira <[email protected]>; pgadmin-hackers
Hi Dave,
On Tue, Jun 5, 2018 at 4:56 PM, Dave Page <[email protected]> wrote:
> Hi
>
> On Tue, Jun 5, 2018 at 9:50 AM, Aditya Toshniwal <aditya.toshniwal@
> enterprisedb.com> wrote:
>
>> Hi Hackers,
>>
>> PFA updated patch. The sqleditor change is sent separately and removed
>> from current patch as suggested.
>> The test cases were running fine when the module was specified using
>> --pkg but were failing in complete run. Fixed that.
>>
>
> I did a quick test by creating a SQL_ASCII database containing a simple
> table:
>
> CREATE TABLE sql_ascii (id serial primary key, data text);
>
> And then populated it with data:
>
> /Library/PostgreSQL/9.4/bin/psql -d sql_ascii -U postgres -c "INSERT INTO
> sql_acsii (data) VALUES ('[Windows-1252] Euro: \x80 Double dagger:
> \x87');"
> /Library/PostgreSQL/9.4/bin/psql -d sql_ascii -U postgres -c "INSERT INTO
> sql_ascii (data) VALUES ('[Latin-1] Yen: \xa5 Half: \xbd');"
> /Library/PostgreSQL/9.4/bin/psql -d sql_ascii -U postgres -c "INSERT INTO
> sql_ascii (data) VALUES ('[Japanese] Ship: \xe8\x88\xb9');"
> /Library/PostgreSQL/9.4/bin/psql -d sql_ascii -U postgres -c "INSERT INTO
> sql_ascii (data) VALUES ('[Invalid UTF-8] Blob: \xf4\xa5\xa3\xa5');"
>
> I then right-clicked the table in the treeview, and selected the option to
> view all rows, and immediately saw an error:
>
> 2018-06-05 12:23:27,319: SQL pgadmin: Execute (async) for server #1 -
> CONN:1187535 (Query-id: 8522474):
> SELECT * FROM public.sql_ascii
> ORDER BY id ASC
> 2018-06-05 12:23:27,320: ERROR pgadmin: Failed to execute query
> (execute_async) for the server #1 - CONN:1187535(Query-id: 8522474):
> Error Message:ERROR: invalid byte sequence for encoding "UTF8": 0x80
> SQL state: 22021
>
> Running "SELECT * FROM sql_ascii" in the query tool resulted in the same
> error, however, if I ran "SET client_encoding = 'SQL_ASCII';" first, I do
> see results.
>
> I have confirmed that I've restarted the server after applying the patch.
>
> What am I missing? Why don't we just set the client_encoding to SQL_ASCII
> if it's a SQL_ASCII database?
>
It is by default same as the server encoding. But, the following existing
code in web/pgadmin/utils/driver/psycopg2/connection.py makes the
client_encoding as UNICODE for every connection. I am not sure it should be
removed.
status = _execute(cur, "SET DateStyle=ISO;"
"SET client_min_messages=notice;"
"SET bytea_output=escape;"
"SET client_encoding='UNICODE';")
Note that this testing was on Python 2.7.10 on MacOS.
>
>
>>
>> Kindly review.
>>
>> Thanks and Regards,
>> Aditya Toshniwal
>> Software Engineer | EnterpriseDB Software Solutions | Pune
>> "Don't Complain about Heat, Plant a tree"
>>
>> On Tue, Jun 5, 2018 at 10:15 AM, Aditya Toshniwal <
>> [email protected]> wrote:
>>
>>> Hi
>>>
>>> On Tue, Jun 5, 2018 at 1:08 AM, Joao De Almeida Pereira <
>>> [email protected]> wrote:
>>>
>>>> Hello Aditya,
>>>>
>>>>>
>>>>>
>>>>> There is no change related to notifications in this patch.
>>>>> The below code is minor fix related to connection status of sql
>>>>> editor. Can you please share the code snippet if it is not the below.
>>>>>
>>>>> - # Check for the asynchronous notifies statements.
>>>>> - conn.check_notifies(True)
>>>>> - notifies = conn.get_notifies()
>>>>> + if status is not None:
>>>>> + # Check for the asynchronous notifies statements.
>>>>> + conn.check_notifies(True)
>>>>> + notifies = conn.get_notifies()
>>>>>
>>>>>
>>>> This is a minor fix, but is it related to querying SQL_ASCII database?
>>>>
>>> No its not. It is something I found when I was working on SQL_ASCII
>>> related changes.
>>> Well then, will send a separate patch for it.
>>>
>>>>
>>>> Thanks
>>>> Victoria && Joao
>>>>
>>>
>>>
>>
>
>
> --
> Dave Page
> Blog: http://pgsnake.blogspot.com
> Twitter: @pgsnake
>
> EnterpriseDB UK: http://www.enterprisedb.com
> The Enterprise PostgreSQL Company
>
^ permalink raw reply [nested|flat] 38+ messages in thread
* Re: [pgAdmin4][RM#3289] Can't query SQL_ASCII database.
@ 2018-06-05 12:35 Dave Page <[email protected]>
parent: Aditya Toshniwal <[email protected]>
0 siblings, 1 reply; 38+ messages in thread
From: Dave Page @ 2018-06-05 12:35 UTC (permalink / raw)
To: Aditya Toshniwal <[email protected]>; +Cc: Joao De Almeida Pereira <[email protected]>; pgadmin-hackers
Hi
On Tue, Jun 5, 2018 at 1:21 PM, Aditya Toshniwal <
[email protected]> wrote:
> Hi Dave,
>
>
> On Tue, Jun 5, 2018 at 4:56 PM, Dave Page <[email protected]> wrote:
>
>> Hi
>>
>> On Tue, Jun 5, 2018 at 9:50 AM, Aditya Toshniwal <
>> [email protected]> wrote:
>>
>>> Hi Hackers,
>>>
>>> PFA updated patch. The sqleditor change is sent separately and removed
>>> from current patch as suggested.
>>> The test cases were running fine when the module was specified using
>>> --pkg but were failing in complete run. Fixed that.
>>>
>>
>> I did a quick test by creating a SQL_ASCII database containing a simple
>> table:
>>
>> CREATE TABLE sql_ascii (id serial primary key, data text);
>>
>> And then populated it with data:
>>
>> /Library/PostgreSQL/9.4/bin/psql -d sql_ascii -U postgres -c "INSERT
>> INTO sql_acsii (data) VALUES ('[Windows-1252] Euro: \x80 Double dagger:
>> \x87');"
>> /Library/PostgreSQL/9.4/bin/psql -d sql_ascii -U postgres -c "INSERT
>> INTO sql_ascii (data) VALUES ('[Latin-1] Yen: \xa5 Half: \xbd');"
>> /Library/PostgreSQL/9.4/bin/psql -d sql_ascii -U postgres -c "INSERT
>> INTO sql_ascii (data) VALUES ('[Japanese] Ship: \xe8\x88\xb9');"
>> /Library/PostgreSQL/9.4/bin/psql -d sql_ascii -U postgres -c "INSERT
>> INTO sql_ascii (data) VALUES ('[Invalid UTF-8] Blob: \xf4\xa5\xa3\xa5');"
>>
>> I then right-clicked the table in the treeview, and selected the option
>> to view all rows, and immediately saw an error:
>>
>> 2018-06-05 12:23:27,319: SQL pgadmin: Execute (async) for server #1 -
>> CONN:1187535 (Query-id: 8522474):
>> SELECT * FROM public.sql_ascii
>> ORDER BY id ASC
>> 2018-06-05 12:23:27,320: ERROR pgadmin: Failed to execute query
>> (execute_async) for the server #1 - CONN:1187535(Query-id: 8522474):
>> Error Message:ERROR: invalid byte sequence for encoding "UTF8": 0x80
>> SQL state: 22021
>>
>> Running "SELECT * FROM sql_ascii" in the query tool resulted in the same
>> error, however, if I ran "SET client_encoding = 'SQL_ASCII';" first, I do
>> see results.
>>
>> I have confirmed that I've restarted the server after applying the patch.
>>
>> What am I missing? Why don't we just set the client_encoding to SQL_ASCII
>> if it's a SQL_ASCII database?
>>
>
> It is by default same as the server encoding. But, the following existing
> code in web/pgadmin/utils/driver/psycopg2/connection.py makes the
> client_encoding as UNICODE for every connection. I am not sure it should be
> removed.
>
> status = _execute(cur, "SET DateStyle=ISO;"
>
> "SET client_min_messages=notice;"
>
> "SET bytea_output=escape;"
>
> "SET client_encoding='UNICODE';")
>
It was probably before you joined, but I have said a number of times that
pgAdmin 3 handled this differently and that maybe we should do it the same
way here. See
https://git.postgresql.org/gitweb/?p=pgadmin3.git;a=blob;f=pgadmin/db/pgConn.cpp,
in the pgConn::Initialize() function.
Either way, your patch isn't working for me.
>
>
> Note that this testing was on Python 2.7.10 on MacOS.
>>
>>
>>>
>>> Kindly review.
>>>
>>> Thanks and Regards,
>>> Aditya Toshniwal
>>> Software Engineer | EnterpriseDB Software Solutions | Pune
>>> "Don't Complain about Heat, Plant a tree"
>>>
>>> On Tue, Jun 5, 2018 at 10:15 AM, Aditya Toshniwal <
>>> [email protected]> wrote:
>>>
>>>> Hi
>>>>
>>>> On Tue, Jun 5, 2018 at 1:08 AM, Joao De Almeida Pereira <
>>>> [email protected]> wrote:
>>>>
>>>>> Hello Aditya,
>>>>>
>>>>>>
>>>>>>
>>>>>> There is no change related to notifications in this patch.
>>>>>> The below code is minor fix related to connection status of sql
>>>>>> editor. Can you please share the code snippet if it is not the below.
>>>>>>
>>>>>> - # Check for the asynchronous notifies statements.
>>>>>> - conn.check_notifies(True)
>>>>>> - notifies = conn.get_notifies()
>>>>>> + if status is not None:
>>>>>> + # Check for the asynchronous notifies statements.
>>>>>> + conn.check_notifies(True)
>>>>>> + notifies = conn.get_notifies()
>>>>>>
>>>>>>
>>>>> This is a minor fix, but is it related to querying SQL_ASCII database?
>>>>>
>>>> No its not. It is something I found when I was working on SQL_ASCII
>>>> related changes.
>>>> Well then, will send a separate patch for it.
>>>>
>>>>>
>>>>> Thanks
>>>>> Victoria && Joao
>>>>>
>>>>
>>>>
>>>
>>
>>
>> --
>> Dave Page
>> Blog: http://pgsnake.blogspot.com
>> Twitter: @pgsnake
>>
>> EnterpriseDB UK: http://www.enterprisedb.com
>> The Enterprise PostgreSQL Company
>>
>
>
--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake
EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 38+ messages in thread
* Re: [pgAdmin4][RM#3289] Can't query SQL_ASCII database.
@ 2018-06-05 12:49 Aditya Toshniwal <[email protected]>
parent: Dave Page <[email protected]>
0 siblings, 2 replies; 38+ messages in thread
From: Aditya Toshniwal @ 2018-06-05 12:49 UTC (permalink / raw)
To: Dave Page <[email protected]>; +Cc: Joao De Almeida Pereira <[email protected]>; pgadmin-hackers
Hi Dave,
The problem of SQL ASCII is solved with the patch, and not related to
setting the client encoding of the sql window. I can see there is no SET
call in pgAdmin3 for client_encoding. I can remove the SET
client_encoding='UNICODE'; that will solve the problem. But, can you please
let me know why that was added.
Will remove the set call and will send you the updated patch if everything
works fine.
Thanks and Regards,
Aditya Toshniwal
Software Engineer | EnterpriseDB Software Solutions | Pune
"Don't Complain about Heat, Plant a tree"
On Tue, Jun 5, 2018 at 6:05 PM, Dave Page <[email protected]> wrote:
> Hi
>
> On Tue, Jun 5, 2018 at 1:21 PM, Aditya Toshniwal <aditya.toshniwal@
> enterprisedb.com> wrote:
>
>> Hi Dave,
>>
>>
>> On Tue, Jun 5, 2018 at 4:56 PM, Dave Page <[email protected]> wrote:
>>
>>> Hi
>>>
>>> On Tue, Jun 5, 2018 at 9:50 AM, Aditya Toshniwal <
>>> [email protected]> wrote:
>>>
>>>> Hi Hackers,
>>>>
>>>> PFA updated patch. The sqleditor change is sent separately and removed
>>>> from current patch as suggested.
>>>> The test cases were running fine when the module was specified using
>>>> --pkg but were failing in complete run. Fixed that.
>>>>
>>>
>>> I did a quick test by creating a SQL_ASCII database containing a simple
>>> table:
>>>
>>> CREATE TABLE sql_ascii (id serial primary key, data text);
>>>
>>> And then populated it with data:
>>>
>>> /Library/PostgreSQL/9.4/bin/psql -d sql_ascii -U postgres -c "INSERT
>>> INTO sql_acsii (data) VALUES ('[Windows-1252] Euro: \x80 Double dagger:
>>> \x87');"
>>> /Library/PostgreSQL/9.4/bin/psql -d sql_ascii -U postgres -c "INSERT
>>> INTO sql_ascii (data) VALUES ('[Latin-1] Yen: \xa5 Half: \xbd');"
>>> /Library/PostgreSQL/9.4/bin/psql -d sql_ascii -U postgres -c "INSERT
>>> INTO sql_ascii (data) VALUES ('[Japanese] Ship: \xe8\x88\xb9');"
>>> /Library/PostgreSQL/9.4/bin/psql -d sql_ascii -U postgres -c "INSERT
>>> INTO sql_ascii (data) VALUES ('[Invalid UTF-8] Blob: \xf4\xa5\xa3\xa5');"
>>>
>>> I then right-clicked the table in the treeview, and selected the option
>>> to view all rows, and immediately saw an error:
>>>
>>> 2018-06-05 12:23:27,319: SQL pgadmin: Execute (async) for server #1 -
>>> CONN:1187535 (Query-id: 8522474):
>>> SELECT * FROM public.sql_ascii
>>> ORDER BY id ASC
>>> 2018-06-05 12:23:27,320: ERROR pgadmin: Failed to execute query
>>> (execute_async) for the server #1 - CONN:1187535(Query-id: 8522474):
>>> Error Message:ERROR: invalid byte sequence for encoding "UTF8": 0x80
>>> SQL state: 22021
>>>
>>> Running "SELECT * FROM sql_ascii" in the query tool resulted in the same
>>> error, however, if I ran "SET client_encoding = 'SQL_ASCII';" first, I do
>>> see results.
>>>
>>> I have confirmed that I've restarted the server after applying the patch.
>>>
>>> What am I missing? Why don't we just set the client_encoding to
>>> SQL_ASCII if it's a SQL_ASCII database?
>>>
>>
>> It is by default same as the server encoding. But, the following existing
>> code in web/pgadmin/utils/driver/psycopg2/connection.py makes the
>> client_encoding as UNICODE for every connection. I am not sure it should be
>> removed.
>>
>> status = _execute(cur, "SET DateStyle=ISO;"
>>
>> "SET client_min_messages=notice;"
>>
>> "SET bytea_output=escape;"
>>
>> "SET client_encoding='UNICODE';")
>>
>
> It was probably before you joined, but I have said a number of times that
> pgAdmin 3 handled this differently and that maybe we should do it the same
> way here. See https://git.postgresql.org/gitweb/?p=pgadmin3.git;a=
> blob;f=pgadmin/db/pgConn.cpp, in the pgConn::Initialize() function.
>
> Either way, your patch isn't working for me.
>
>
>>
>>
>> Note that this testing was on Python 2.7.10 on MacOS.
>>>
>>>
>>>>
>>>> Kindly review.
>>>>
>>>> Thanks and Regards,
>>>> Aditya Toshniwal
>>>> Software Engineer | EnterpriseDB Software Solutions | Pune
>>>> "Don't Complain about Heat, Plant a tree"
>>>>
>>>> On Tue, Jun 5, 2018 at 10:15 AM, Aditya Toshniwal <
>>>> [email protected]> wrote:
>>>>
>>>>> Hi
>>>>>
>>>>> On Tue, Jun 5, 2018 at 1:08 AM, Joao De Almeida Pereira <
>>>>> [email protected]> wrote:
>>>>>
>>>>>> Hello Aditya,
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> There is no change related to notifications in this patch.
>>>>>>> The below code is minor fix related to connection status of sql
>>>>>>> editor. Can you please share the code snippet if it is not the below.
>>>>>>>
>>>>>>> - # Check for the asynchronous notifies statements.
>>>>>>> - conn.check_notifies(True)
>>>>>>> - notifies = conn.get_notifies()
>>>>>>> + if status is not None:
>>>>>>> + # Check for the asynchronous notifies statements.
>>>>>>> + conn.check_notifies(True)
>>>>>>> + notifies = conn.get_notifies()
>>>>>>>
>>>>>>>
>>>>>> This is a minor fix, but is it related to querying SQL_ASCII database?
>>>>>>
>>>>> No its not. It is something I found when I was working on SQL_ASCII
>>>>> related changes.
>>>>> Well then, will send a separate patch for it.
>>>>>
>>>>>>
>>>>>> Thanks
>>>>>> Victoria && Joao
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>> --
>>> Dave Page
>>> Blog: http://pgsnake.blogspot.com
>>> Twitter: @pgsnake
>>>
>>> EnterpriseDB UK: http://www.enterprisedb.com
>>> The Enterprise PostgreSQL Company
>>>
>>
>>
>
>
> --
> Dave Page
> Blog: http://pgsnake.blogspot.com
> Twitter: @pgsnake
>
> EnterpriseDB UK: http://www.enterprisedb.com
> The Enterprise PostgreSQL Company
>
^ permalink raw reply [nested|flat] 38+ messages in thread
* Re: [pgAdmin4][RM#3289] Can't query SQL_ASCII database.
@ 2018-06-05 12:53 Aditya Toshniwal <[email protected]>
parent: Aditya Toshniwal <[email protected]>
1 sibling, 0 replies; 38+ messages in thread
From: Aditya Toshniwal @ 2018-06-05 12:53 UTC (permalink / raw)
To: Dave Page <[email protected]>; +Cc: Joao De Almeida Pereira <[email protected]>; pgadmin-hackers
Also, there are limitations in psycopg2 as well related to encoding, which
makes it even more difficult to make behaviour similar to pgAdmin3 as there
are encodings and decodings done at pyscopg2 end also.
Thanks and Regards,
Aditya Toshniwal
Software Engineer | EnterpriseDB Software Solutions | Pune
"Don't Complain about Heat, Plant a tree"
On Tue, Jun 5, 2018 at 6:19 PM, Aditya Toshniwal <
[email protected]> wrote:
> Hi Dave,
>
> The problem of SQL ASCII is solved with the patch, and not related to
> setting the client encoding of the sql window. I can see there is no SET
> call in pgAdmin3 for client_encoding. I can remove the SET
> client_encoding='UNICODE'; that will solve the problem. But, can you
> please let me know why that was added.
>
> Will remove the set call and will send you the updated patch if everything
> works fine.
>
>
> Thanks and Regards,
> Aditya Toshniwal
> Software Engineer | EnterpriseDB Software Solutions | Pune
> "Don't Complain about Heat, Plant a tree"
>
> On Tue, Jun 5, 2018 at 6:05 PM, Dave Page <[email protected]> wrote:
>
>> Hi
>>
>> On Tue, Jun 5, 2018 at 1:21 PM, Aditya Toshniwal <
>> [email protected]> wrote:
>>
>>> Hi Dave,
>>>
>>>
>>> On Tue, Jun 5, 2018 at 4:56 PM, Dave Page <[email protected]> wrote:
>>>
>>>> Hi
>>>>
>>>> On Tue, Jun 5, 2018 at 9:50 AM, Aditya Toshniwal <
>>>> [email protected]> wrote:
>>>>
>>>>> Hi Hackers,
>>>>>
>>>>> PFA updated patch. The sqleditor change is sent separately and removed
>>>>> from current patch as suggested.
>>>>> The test cases were running fine when the module was specified using
>>>>> --pkg but were failing in complete run. Fixed that.
>>>>>
>>>>
>>>> I did a quick test by creating a SQL_ASCII database containing a simple
>>>> table:
>>>>
>>>> CREATE TABLE sql_ascii (id serial primary key, data text);
>>>>
>>>> And then populated it with data:
>>>>
>>>> /Library/PostgreSQL/9.4/bin/psql -d sql_ascii -U postgres -c "INSERT
>>>> INTO sql_acsii (data) VALUES ('[Windows-1252] Euro: \x80 Double dagger:
>>>> \x87');"
>>>> /Library/PostgreSQL/9.4/bin/psql -d sql_ascii -U postgres -c "INSERT
>>>> INTO sql_ascii (data) VALUES ('[Latin-1] Yen: \xa5 Half: \xbd');"
>>>> /Library/PostgreSQL/9.4/bin/psql -d sql_ascii -U postgres -c "INSERT
>>>> INTO sql_ascii (data) VALUES ('[Japanese] Ship: \xe8\x88\xb9');"
>>>> /Library/PostgreSQL/9.4/bin/psql -d sql_ascii -U postgres -c "INSERT
>>>> INTO sql_ascii (data) VALUES ('[Invalid UTF-8] Blob: \xf4\xa5\xa3\xa5');"
>>>>
>>>> I then right-clicked the table in the treeview, and selected the option
>>>> to view all rows, and immediately saw an error:
>>>>
>>>> 2018-06-05 12:23:27,319: SQL pgadmin: Execute (async) for server #1 -
>>>> CONN:1187535 (Query-id: 8522474):
>>>> SELECT * FROM public.sql_ascii
>>>> ORDER BY id ASC
>>>> 2018-06-05 12:23:27,320: ERROR pgadmin: Failed to execute query
>>>> (execute_async) for the server #1 - CONN:1187535(Query-id: 8522474):
>>>> Error Message:ERROR: invalid byte sequence for encoding "UTF8": 0x80
>>>> SQL state: 22021
>>>>
>>>> Running "SELECT * FROM sql_ascii" in the query tool resulted in the
>>>> same error, however, if I ran "SET client_encoding = 'SQL_ASCII';" first, I
>>>> do see results.
>>>>
>>>> I have confirmed that I've restarted the server after applying the
>>>> patch.
>>>>
>>>> What am I missing? Why don't we just set the client_encoding to
>>>> SQL_ASCII if it's a SQL_ASCII database?
>>>>
>>>
>>> It is by default same as the server encoding. But, the following
>>> existing code in web/pgadmin/utils/driver/psycopg2/connection.py makes
>>> the client_encoding as UNICODE for every connection. I am not sure it
>>> should be removed.
>>>
>>> status = _execute(cur, "SET DateStyle=ISO;"
>>>
>>> "SET client_min_messages=notice;"
>>>
>>> "SET bytea_output=escape;"
>>>
>>> "SET client_encoding='UNICODE';")
>>>
>>
>> It was probably before you joined, but I have said a number of times that
>> pgAdmin 3 handled this differently and that maybe we should do it the same
>> way here. See https://git.postgresql.org/gitweb/?p=pgadmin3.git;a=blob
>> ;f=pgadmin/db/pgConn.cpp, in the pgConn::Initialize() function.
>>
>> Either way, your patch isn't working for me.
>>
>>
>>>
>>>
>>> Note that this testing was on Python 2.7.10 on MacOS.
>>>>
>>>>
>>>>>
>>>>> Kindly review.
>>>>>
>>>>> Thanks and Regards,
>>>>> Aditya Toshniwal
>>>>> Software Engineer | EnterpriseDB Software Solutions | Pune
>>>>> "Don't Complain about Heat, Plant a tree"
>>>>>
>>>>> On Tue, Jun 5, 2018 at 10:15 AM, Aditya Toshniwal <
>>>>> [email protected]> wrote:
>>>>>
>>>>>> Hi
>>>>>>
>>>>>> On Tue, Jun 5, 2018 at 1:08 AM, Joao De Almeida Pereira <
>>>>>> [email protected]> wrote:
>>>>>>
>>>>>>> Hello Aditya,
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> There is no change related to notifications in this patch.
>>>>>>>> The below code is minor fix related to connection status of sql
>>>>>>>> editor. Can you please share the code snippet if it is not the below.
>>>>>>>>
>>>>>>>> - # Check for the asynchronous notifies statements.
>>>>>>>> - conn.check_notifies(True)
>>>>>>>> - notifies = conn.get_notifies()
>>>>>>>> + if status is not None:
>>>>>>>> + # Check for the asynchronous notifies statements.
>>>>>>>> + conn.check_notifies(True)
>>>>>>>> + notifies = conn.get_notifies()
>>>>>>>>
>>>>>>>>
>>>>>>> This is a minor fix, but is it related to querying SQL_ASCII
>>>>>>> database?
>>>>>>>
>>>>>> No its not. It is something I found when I was working on SQL_ASCII
>>>>>> related changes.
>>>>>> Well then, will send a separate patch for it.
>>>>>>
>>>>>>>
>>>>>>> Thanks
>>>>>>> Victoria && Joao
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Dave Page
>>>> Blog: http://pgsnake.blogspot.com
>>>> Twitter: @pgsnake
>>>>
>>>> EnterpriseDB UK: http://www.enterprisedb.com
>>>> The Enterprise PostgreSQL Company
>>>>
>>>
>>>
>>
>>
>> --
>> Dave Page
>> Blog: http://pgsnake.blogspot.com
>> Twitter: @pgsnake
>>
>> EnterpriseDB UK: http://www.enterprisedb.com
>> The Enterprise PostgreSQL Company
>>
>
>
^ permalink raw reply [nested|flat] 38+ messages in thread
* Re: [pgAdmin4][RM#3289] Can't query SQL_ASCII database.
@ 2018-06-05 12:55 Dave Page <[email protected]>
parent: Aditya Toshniwal <[email protected]>
1 sibling, 1 reply; 38+ messages in thread
From: Dave Page @ 2018-06-05 12:55 UTC (permalink / raw)
To: Aditya Toshniwal <[email protected]>; +Cc: Joao De Almeida Pereira <[email protected]>; pgadmin-hackers
On Tue, Jun 5, 2018 at 1:49 PM, Aditya Toshniwal <
[email protected]> wrote:
> Hi Dave,
>
> The problem of SQL ASCII is solved with the patch, and not related to
> setting the client encoding of the sql window.
>
No it's not. It doesn't work for me as I said (and showed the example of).
> I can see there is no SET call in pgAdmin3 for client_encoding. I can
> remove the SET client_encoding='UNICODE'; that will solve the problem.
> But, can you please let me know why that was added.
>
There is, but it's inside an API call (PQsetClientEncoding):
300
<https://git.postgresql.org/gitweb/?p=pgadmin3.git;a=blob;f=pgadmin/db/pgConn.cpp#l300;
wxLogInfo(wxT("Setting client_encoding to '%s'"), encoding.c_str());
301
<https://git.postgresql.org/gitweb/?p=pgadmin3.git;a=blob;f=pgadmin/db/pgConn.cpp#l301;
if (PQsetClientEncoding(conn, encoding.ToAscii()))
302
<https://git.postgresql.org/gitweb/?p=pgadmin3.git;a=blob;f=pgadmin/db/pgConn.cpp#l302;
{
303
<https://git.postgresql.org/gitweb/?p=pgadmin3.git;a=blob;f=pgadmin/db/pgConn.cpp#l303;
wxLogError(wxT("%s"), GetLastError().c_str());
304
<https://git.postgresql.org/gitweb/?p=pgadmin3.git;a=blob;f=pgadmin/db/pgConn.cpp#l304;
}
305
<https://git.postgresql.org/gitweb/?p=pgadmin3.git;a=blob;f=pgadmin/db/pgConn.cpp#l305;
>
> Will remove the set call and will send you the updated patch if everything
> works fine.
>
No, we need to ensure the client encoding is set correctly. It just needs
to be set to SQL_ASCII if it's a SQL_ASCII database (I believe).
>
>
> Thanks and Regards,
> Aditya Toshniwal
> Software Engineer | EnterpriseDB Software Solutions | Pune
> "Don't Complain about Heat, Plant a tree"
>
> On Tue, Jun 5, 2018 at 6:05 PM, Dave Page <[email protected]> wrote:
>
>> Hi
>>
>> On Tue, Jun 5, 2018 at 1:21 PM, Aditya Toshniwal <
>> [email protected]> wrote:
>>
>>> Hi Dave,
>>>
>>>
>>> On Tue, Jun 5, 2018 at 4:56 PM, Dave Page <[email protected]> wrote:
>>>
>>>> Hi
>>>>
>>>> On Tue, Jun 5, 2018 at 9:50 AM, Aditya Toshniwal <
>>>> [email protected]> wrote:
>>>>
>>>>> Hi Hackers,
>>>>>
>>>>> PFA updated patch. The sqleditor change is sent separately and removed
>>>>> from current patch as suggested.
>>>>> The test cases were running fine when the module was specified using
>>>>> --pkg but were failing in complete run. Fixed that.
>>>>>
>>>>
>>>> I did a quick test by creating a SQL_ASCII database containing a simple
>>>> table:
>>>>
>>>> CREATE TABLE sql_ascii (id serial primary key, data text);
>>>>
>>>> And then populated it with data:
>>>>
>>>> /Library/PostgreSQL/9.4/bin/psql -d sql_ascii -U postgres -c "INSERT
>>>> INTO sql_acsii (data) VALUES ('[Windows-1252] Euro: \x80 Double dagger:
>>>> \x87');"
>>>> /Library/PostgreSQL/9.4/bin/psql -d sql_ascii -U postgres -c "INSERT
>>>> INTO sql_ascii (data) VALUES ('[Latin-1] Yen: \xa5 Half: \xbd');"
>>>> /Library/PostgreSQL/9.4/bin/psql -d sql_ascii -U postgres -c "INSERT
>>>> INTO sql_ascii (data) VALUES ('[Japanese] Ship: \xe8\x88\xb9');"
>>>> /Library/PostgreSQL/9.4/bin/psql -d sql_ascii -U postgres -c "INSERT
>>>> INTO sql_ascii (data) VALUES ('[Invalid UTF-8] Blob: \xf4\xa5\xa3\xa5');"
>>>>
>>>> I then right-clicked the table in the treeview, and selected the option
>>>> to view all rows, and immediately saw an error:
>>>>
>>>> 2018-06-05 12:23:27,319: SQL pgadmin: Execute (async) for server #1 -
>>>> CONN:1187535 (Query-id: 8522474):
>>>> SELECT * FROM public.sql_ascii
>>>> ORDER BY id ASC
>>>> 2018-06-05 12:23:27,320: ERROR pgadmin: Failed to execute query
>>>> (execute_async) for the server #1 - CONN:1187535(Query-id: 8522474):
>>>> Error Message:ERROR: invalid byte sequence for encoding "UTF8": 0x80
>>>> SQL state: 22021
>>>>
>>>> Running "SELECT * FROM sql_ascii" in the query tool resulted in the
>>>> same error, however, if I ran "SET client_encoding = 'SQL_ASCII';" first, I
>>>> do see results.
>>>>
>>>> I have confirmed that I've restarted the server after applying the
>>>> patch.
>>>>
>>>> What am I missing? Why don't we just set the client_encoding to
>>>> SQL_ASCII if it's a SQL_ASCII database?
>>>>
>>>
>>> It is by default same as the server encoding. But, the following
>>> existing code in web/pgadmin/utils/driver/psycopg2/connection.py makes
>>> the client_encoding as UNICODE for every connection. I am not sure it
>>> should be removed.
>>>
>>> status = _execute(cur, "SET DateStyle=ISO;"
>>>
>>> "SET client_min_messages=notice;"
>>>
>>> "SET bytea_output=escape;"
>>>
>>> "SET client_encoding='UNICODE';")
>>>
>>
>> It was probably before you joined, but I have said a number of times that
>> pgAdmin 3 handled this differently and that maybe we should do it the same
>> way here. See https://git.postgresql.org/gitweb/?p=pgadmin3.git;a=blob
>> ;f=pgadmin/db/pgConn.cpp, in the pgConn::Initialize() function.
>>
>> Either way, your patch isn't working for me.
>>
>>
>>>
>>>
>>> Note that this testing was on Python 2.7.10 on MacOS.
>>>>
>>>>
>>>>>
>>>>> Kindly review.
>>>>>
>>>>> Thanks and Regards,
>>>>> Aditya Toshniwal
>>>>> Software Engineer | EnterpriseDB Software Solutions | Pune
>>>>> "Don't Complain about Heat, Plant a tree"
>>>>>
>>>>> On Tue, Jun 5, 2018 at 10:15 AM, Aditya Toshniwal <
>>>>> [email protected]> wrote:
>>>>>
>>>>>> Hi
>>>>>>
>>>>>> On Tue, Jun 5, 2018 at 1:08 AM, Joao De Almeida Pereira <
>>>>>> [email protected]> wrote:
>>>>>>
>>>>>>> Hello Aditya,
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> There is no change related to notifications in this patch.
>>>>>>>> The below code is minor fix related to connection status of sql
>>>>>>>> editor. Can you please share the code snippet if it is not the below.
>>>>>>>>
>>>>>>>> - # Check for the asynchronous notifies statements.
>>>>>>>> - conn.check_notifies(True)
>>>>>>>> - notifies = conn.get_notifies()
>>>>>>>> + if status is not None:
>>>>>>>> + # Check for the asynchronous notifies statements.
>>>>>>>> + conn.check_notifies(True)
>>>>>>>> + notifies = conn.get_notifies()
>>>>>>>>
>>>>>>>>
>>>>>>> This is a minor fix, but is it related to querying SQL_ASCII
>>>>>>> database?
>>>>>>>
>>>>>> No its not. It is something I found when I was working on SQL_ASCII
>>>>>> related changes.
>>>>>> Well then, will send a separate patch for it.
>>>>>>
>>>>>>>
>>>>>>> Thanks
>>>>>>> Victoria && Joao
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Dave Page
>>>> Blog: http://pgsnake.blogspot.com
>>>> Twitter: @pgsnake
>>>>
>>>> EnterpriseDB UK: http://www.enterprisedb.com
>>>> The Enterprise PostgreSQL Company
>>>>
>>>
>>>
>>
>>
>> --
>> Dave Page
>> Blog: http://pgsnake.blogspot.com
>> Twitter: @pgsnake
>>
>> EnterpriseDB UK: http://www.enterprisedb.com
>> The Enterprise PostgreSQL Company
>>
>
>
--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake
EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 38+ messages in thread
* Re: [pgAdmin4][RM#3289] Can't query SQL_ASCII database.
@ 2018-06-05 13:03 Aditya Toshniwal <[email protected]>
parent: Dave Page <[email protected]>
0 siblings, 1 reply; 38+ messages in thread
From: Aditya Toshniwal @ 2018-06-05 13:03 UTC (permalink / raw)
To: Dave Page <[email protected]>; +Cc: Joao De Almeida Pereira <[email protected]>; pgadmin-hackers
Hi
On Tue, Jun 5, 2018 at 6:25 PM, Dave Page <[email protected]> wrote:
>
>
> On Tue, Jun 5, 2018 at 1:49 PM, Aditya Toshniwal <aditya.toshniwal@
> enterprisedb.com> wrote:
>
>> Hi Dave,
>>
>> The problem of SQL ASCII is solved with the patch, and not related to
>> setting the client encoding of the sql window.
>>
>
> No it's not. It doesn't work for me as I said (and showed the example of).
>
After setting the client_encoding to SQL_ASCII you got the output.
Previously, it used to fail in the back end itself because python encoding
failure. That is fixed.
The error ERROR: invalid byte sequence for encoding "UTF8": 0x80 is thrown
by postgres and not python or pgAdmin4. You will get the same error even if
you
connect from psql.
>
>
>> I can see there is no SET call in pgAdmin3 for client_encoding. I can
>> remove the SET client_encoding='UNICODE'; that will solve the problem.
>> But, can you please let me know why that was added.
>>
>
> There is, but it's inside an API call (PQsetClientEncoding):
>
> 300
> <https://git.postgresql.org/gitweb/?p=pgadmin3.git;a=blob;f=pgadmin/db/pgConn.cpp#l300;
> wxLogInfo(wxT("Setting client_encoding to '%
> s'"), encoding.c_str());
> 301
> <https://git.postgresql.org/gitweb/?p=pgadmin3.git;a=blob;f=pgadmin/db/pgConn.cpp#l301;
> if (PQsetClientEncoding(conn, encoding.ToAscii()))
> 302
> <https://git.postgresql.org/gitweb/?p=pgadmin3.git;a=blob;f=pgadmin/db/pgConn.cpp#l302;
> {
> 303
> <https://git.postgresql.org/gitweb/?p=pgadmin3.git;a=blob;f=pgadmin/db/pgConn.cpp#l303;
> wxLogError(wxT("%s"), GetLastError().c_str());
> 304
> <https://git.postgresql.org/gitweb/?p=pgadmin3.git;a=blob;f=pgadmin/db/pgConn.cpp#l304;
> }
> 305
> <https://git.postgresql.org/gitweb/?p=pgadmin3.git;a=blob;f=pgadmin/db/pgConn.cpp#l305;
>
> Oops ! Missed that. Apologies.
>
>
>>
>> Will remove the set call and will send you the updated patch if
>> everything works fine.
>>
>
> No, we need to ensure the client encoding is set correctly. It just needs
> to be set to SQL_ASCII if it's a SQL_ASCII database (I believe).
>
>
Need to rework on the initialise method. Will come with an updated. patch.
Sorry for trouble.
>
>>
>> On Tue, Jun 5, 2018 at 6:05 PM, Dave Page <[email protected]> wrote:
>>
>>> Hi
>>>
>>> On Tue, Jun 5, 2018 at 1:21 PM, Aditya Toshniwal <
>>> [email protected]> wrote:
>>>
>>>> Hi Dave,
>>>>
>>>>
>>>> On Tue, Jun 5, 2018 at 4:56 PM, Dave Page <[email protected]> wrote:
>>>>
>>>>> Hi
>>>>>
>>>>> On Tue, Jun 5, 2018 at 9:50 AM, Aditya Toshniwal <
>>>>> [email protected]> wrote:
>>>>>
>>>>>> Hi Hackers,
>>>>>>
>>>>>> PFA updated patch. The sqleditor change is sent separately and
>>>>>> removed from current patch as suggested.
>>>>>> The test cases were running fine when the module was specified using
>>>>>> --pkg but were failing in complete run. Fixed that.
>>>>>>
>>>>>
>>>>> I did a quick test by creating a SQL_ASCII database containing a
>>>>> simple table:
>>>>>
>>>>> CREATE TABLE sql_ascii (id serial primary key, data text);
>>>>>
>>>>> And then populated it with data:
>>>>>
>>>>> /Library/PostgreSQL/9.4/bin/psql -d sql_ascii -U postgres -c "INSERT
>>>>> INTO sql_acsii (data) VALUES ('[Windows-1252] Euro: \x80 Double dagger:
>>>>> \x87');"
>>>>> /Library/PostgreSQL/9.4/bin/psql -d sql_ascii -U postgres -c "INSERT
>>>>> INTO sql_ascii (data) VALUES ('[Latin-1] Yen: \xa5 Half: \xbd');"
>>>>> /Library/PostgreSQL/9.4/bin/psql -d sql_ascii -U postgres -c "INSERT
>>>>> INTO sql_ascii (data) VALUES ('[Japanese] Ship: \xe8\x88\xb9');"
>>>>> /Library/PostgreSQL/9.4/bin/psql -d sql_ascii -U postgres -c "INSERT
>>>>> INTO sql_ascii (data) VALUES ('[Invalid UTF-8] Blob: \xf4\xa5\xa3\xa5');"
>>>>>
>>>>> I then right-clicked the table in the treeview, and selected the
>>>>> option to view all rows, and immediately saw an error:
>>>>>
>>>>> 2018-06-05 12:23:27,319: SQL pgadmin: Execute (async) for server #1 -
>>>>> CONN:1187535 (Query-id: 8522474):
>>>>> SELECT * FROM public.sql_ascii
>>>>> ORDER BY id ASC
>>>>> 2018-06-05 12:23:27,320: ERROR pgadmin: Failed to execute query
>>>>> (execute_async) for the server #1 - CONN:1187535(Query-id: 8522474):
>>>>> Error Message:ERROR: invalid byte sequence for encoding "UTF8": 0x80
>>>>> SQL state: 22021
>>>>>
>>>>> Running "SELECT * FROM sql_ascii" in the query tool resulted in the
>>>>> same error, however, if I ran "SET client_encoding = 'SQL_ASCII';" first, I
>>>>> do see results.
>>>>>
>>>>> I have confirmed that I've restarted the server after applying the
>>>>> patch.
>>>>>
>>>>> What am I missing? Why don't we just set the client_encoding to
>>>>> SQL_ASCII if it's a SQL_ASCII database?
>>>>>
>>>>
>>>> It is by default same as the server encoding. But, the following
>>>> existing code in web/pgadmin/utils/driver/psycopg2/connection.py makes
>>>> the client_encoding as UNICODE for every connection. I am not sure it
>>>> should be removed.
>>>>
>>>> status = _execute(cur, "SET DateStyle=ISO;"
>>>>
>>>> "SET client_min_messages=notice;"
>>>>
>>>> "SET bytea_output=escape;"
>>>>
>>>> "SET client_encoding='UNICODE';")
>>>>
>>>
>>> It was probably before you joined, but I have said a number of times
>>> that pgAdmin 3 handled this differently and that maybe we should do it the
>>> same way here. See https://git.postgresql.org
>>> /gitweb/?p=pgadmin3.git;a=blob;f=pgadmin/db/pgConn.cpp, in the
>>> pgConn::Initialize() function.
>>>
>>> Either way, your patch isn't working for me.
>>>
>>>
>>>>
>>>>
>>>> Note that this testing was on Python 2.7.10 on MacOS.
>>>>>
>>>>>
>>>>>>
>>>>>> Kindly review.
>>>>>>
>>>>>> Thanks and Regards,
>>>>>> Aditya Toshniwal
>>>>>> Software Engineer | EnterpriseDB Software Solutions | Pune
>>>>>> "Don't Complain about Heat, Plant a tree"
>>>>>>
>>>>>> On Tue, Jun 5, 2018 at 10:15 AM, Aditya Toshniwal <
>>>>>> [email protected]> wrote:
>>>>>>
>>>>>>> Hi
>>>>>>>
>>>>>>> On Tue, Jun 5, 2018 at 1:08 AM, Joao De Almeida Pereira <
>>>>>>> [email protected]> wrote:
>>>>>>>
>>>>>>>> Hello Aditya,
>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> There is no change related to notifications in this patch.
>>>>>>>>> The below code is minor fix related to connection status of sql
>>>>>>>>> editor. Can you please share the code snippet if it is not the below.
>>>>>>>>>
>>>>>>>>> - # Check for the asynchronous notifies statements.
>>>>>>>>> - conn.check_notifies(True)
>>>>>>>>> - notifies = conn.get_notifies()
>>>>>>>>> + if status is not None:
>>>>>>>>> + # Check for the asynchronous notifies statements.
>>>>>>>>> + conn.check_notifies(True)
>>>>>>>>> + notifies = conn.get_notifies()
>>>>>>>>>
>>>>>>>>>
>>>>>>>> This is a minor fix, but is it related to querying SQL_ASCII
>>>>>>>> database?
>>>>>>>>
>>>>>>> No its not. It is something I found when I was working on SQL_ASCII
>>>>>>> related changes.
>>>>>>> Well then, will send a separate patch for it.
>>>>>>>
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>> Victoria && Joao
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Dave Page
>>>>> Blog: http://pgsnake.blogspot.com
>>>>> Twitter: @pgsnake
>>>>>
>>>>> EnterpriseDB UK: http://www.enterprisedb.com
>>>>> The Enterprise PostgreSQL Company
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Dave Page
>>> Blog: http://pgsnake.blogspot.com
>>> Twitter: @pgsnake
>>>
>>> EnterpriseDB UK: http://www.enterprisedb.com
>>> The Enterprise PostgreSQL Company
>>>
>>
>>
>
>
> --
> Dave Page
> Blog: http://pgsnake.blogspot.com
> Twitter: @pgsnake
>
> EnterpriseDB UK: http://www.enterprisedb.com
> The Enterprise PostgreSQL Company
>
^ permalink raw reply [nested|flat] 38+ messages in thread
* Re: [pgAdmin4][RM#3289] Can't query SQL_ASCII database.
@ 2018-06-05 13:12 Dave Page <[email protected]>
parent: Aditya Toshniwal <[email protected]>
0 siblings, 1 reply; 38+ messages in thread
From: Dave Page @ 2018-06-05 13:12 UTC (permalink / raw)
To: Aditya Toshniwal <[email protected]>; +Cc: Joao De Almeida Pereira <[email protected]>; pgadmin-hackers
Hi
On Tue, Jun 5, 2018 at 2:03 PM, Aditya Toshniwal <
[email protected]> wrote:
> Hi
>
> On Tue, Jun 5, 2018 at 6:25 PM, Dave Page <[email protected]> wrote:
>
>>
>>
>> On Tue, Jun 5, 2018 at 1:49 PM, Aditya Toshniwal <
>> [email protected]> wrote:
>>
>>> Hi Dave,
>>>
>>> The problem of SQL ASCII is solved with the patch, and not related to
>>> setting the client encoding of the sql window.
>>>
>>
>> No it's not. It doesn't work for me as I said (and showed the example of).
>>
>
> After setting the client_encoding to SQL_ASCII you got the output.
> Previously, it used to fail in the back end itself because python encoding
> failure. That is fixed.
> The error ERROR: invalid byte sequence for encoding "UTF8": 0x80 is
> thrown by postgres and not python or pgAdmin4. You will get the same error
> even if you
> connect from psql.
>
Sure - but that is not a fix. You have no way of running the SET command if
you're using "view data" - and in the query tool, users just expect it to
work (as it did in pgAdmin 3).
>
>>
>>> I can see there is no SET call in pgAdmin3 for client_encoding. I can
>>> remove the SET client_encoding='UNICODE'; that will solve the problem.
>>> But, can you please let me know why that was added.
>>>
>>
>> There is, but it's inside an API call (PQsetClientEncoding):
>>
>> 300
>> <https://git.postgresql.org/gitweb/?p=pgadmin3.git;a=blob;f=pgadmin/db/pgConn.cpp#l300;
>> wxLogInfo(wxT("Setting client_encoding to '%s'")
>> , encoding.c_str());
>> 301
>> <https://git.postgresql.org/gitweb/?p=pgadmin3.git;a=blob;f=pgadmin/db/pgConn.cpp#l301;
>> if (PQsetClientEncoding(conn, encoding.ToAscii()))
>> 302
>> <https://git.postgresql.org/gitweb/?p=pgadmin3.git;a=blob;f=pgadmin/db/pgConn.cpp#l302;
>> {
>> 303
>> <https://git.postgresql.org/gitweb/?p=pgadmin3.git;a=blob;f=pgadmin/db/pgConn.cpp#l303;
>> wxLogError(wxT("%s"), GetLastError().c_str());
>> 304
>> <https://git.postgresql.org/gitweb/?p=pgadmin3.git;a=blob;f=pgadmin/db/pgConn.cpp#l304;
>> }
>> 305
>> <https://git.postgresql.org/gitweb/?p=pgadmin3.git;a=blob;f=pgadmin/db/pgConn.cpp#l305;
>>
>> Oops ! Missed that. Apologies.
>
>>
>>
>>>
>>> Will remove the set call and will send you the updated patch if
>>> everything works fine.
>>>
>>
>> No, we need to ensure the client encoding is set correctly. It just needs
>> to be set to SQL_ASCII if it's a SQL_ASCII database (I believe).
>>
>>
> Need to rework on the initialise method. Will come with an updated. patch.
> Sorry for trouble.
>
>>
>>>
>>> On Tue, Jun 5, 2018 at 6:05 PM, Dave Page <[email protected]> wrote:
>>>
>>>> Hi
>>>>
>>>> On Tue, Jun 5, 2018 at 1:21 PM, Aditya Toshniwal <
>>>> [email protected]> wrote:
>>>>
>>>>> Hi Dave,
>>>>>
>>>>>
>>>>> On Tue, Jun 5, 2018 at 4:56 PM, Dave Page <[email protected]> wrote:
>>>>>
>>>>>> Hi
>>>>>>
>>>>>> On Tue, Jun 5, 2018 at 9:50 AM, Aditya Toshniwal <
>>>>>> [email protected]> wrote:
>>>>>>
>>>>>>> Hi Hackers,
>>>>>>>
>>>>>>> PFA updated patch. The sqleditor change is sent separately and
>>>>>>> removed from current patch as suggested.
>>>>>>> The test cases were running fine when the module was specified using
>>>>>>> --pkg but were failing in complete run. Fixed that.
>>>>>>>
>>>>>>
>>>>>> I did a quick test by creating a SQL_ASCII database containing a
>>>>>> simple table:
>>>>>>
>>>>>> CREATE TABLE sql_ascii (id serial primary key, data text);
>>>>>>
>>>>>> And then populated it with data:
>>>>>>
>>>>>> /Library/PostgreSQL/9.4/bin/psql -d sql_ascii -U postgres -c "INSERT
>>>>>> INTO sql_acsii (data) VALUES ('[Windows-1252] Euro: \x80 Double dagger:
>>>>>> \x87');"
>>>>>> /Library/PostgreSQL/9.4/bin/psql -d sql_ascii -U postgres -c "INSERT
>>>>>> INTO sql_ascii (data) VALUES ('[Latin-1] Yen: \xa5 Half: \xbd');"
>>>>>> /Library/PostgreSQL/9.4/bin/psql -d sql_ascii -U postgres -c "INSERT
>>>>>> INTO sql_ascii (data) VALUES ('[Japanese] Ship: \xe8\x88\xb9');"
>>>>>> /Library/PostgreSQL/9.4/bin/psql -d sql_ascii -U postgres -c "INSERT
>>>>>> INTO sql_ascii (data) VALUES ('[Invalid UTF-8] Blob: \xf4\xa5\xa3\xa5');"
>>>>>>
>>>>>> I then right-clicked the table in the treeview, and selected the
>>>>>> option to view all rows, and immediately saw an error:
>>>>>>
>>>>>> 2018-06-05 12:23:27,319: SQL pgadmin: Execute (async) for server #1
>>>>>> - CONN:1187535 (Query-id: 8522474):
>>>>>> SELECT * FROM public.sql_ascii
>>>>>> ORDER BY id ASC
>>>>>> 2018-06-05 12:23:27,320: ERROR pgadmin: Failed to execute query
>>>>>> (execute_async) for the server #1 - CONN:1187535(Query-id: 8522474):
>>>>>> Error Message:ERROR: invalid byte sequence for encoding "UTF8": 0x80
>>>>>> SQL state: 22021
>>>>>>
>>>>>> Running "SELECT * FROM sql_ascii" in the query tool resulted in the
>>>>>> same error, however, if I ran "SET client_encoding = 'SQL_ASCII';" first, I
>>>>>> do see results.
>>>>>>
>>>>>> I have confirmed that I've restarted the server after applying the
>>>>>> patch.
>>>>>>
>>>>>> What am I missing? Why don't we just set the client_encoding to
>>>>>> SQL_ASCII if it's a SQL_ASCII database?
>>>>>>
>>>>>
>>>>> It is by default same as the server encoding. But, the following
>>>>> existing code in web/pgadmin/utils/driver/psycopg2/connection.py makes
>>>>> the client_encoding as UNICODE for every connection. I am not sure it
>>>>> should be removed.
>>>>>
>>>>> status = _execute(cur, "SET DateStyle=ISO;"
>>>>>
>>>>> "SET client_min_messages=notice;"
>>>>>
>>>>> "SET bytea_output=escape;"
>>>>>
>>>>> "SET client_encoding='UNICODE';")
>>>>>
>>>>
>>>> It was probably before you joined, but I have said a number of times
>>>> that pgAdmin 3 handled this differently and that maybe we should do it the
>>>> same way here. See https://git.postgresql.org
>>>> /gitweb/?p=pgadmin3.git;a=blob;f=pgadmin/db/pgConn.cpp, in the
>>>> pgConn::Initialize() function.
>>>>
>>>> Either way, your patch isn't working for me.
>>>>
>>>>
>>>>>
>>>>>
>>>>> Note that this testing was on Python 2.7.10 on MacOS.
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> Kindly review.
>>>>>>>
>>>>>>> Thanks and Regards,
>>>>>>> Aditya Toshniwal
>>>>>>> Software Engineer | EnterpriseDB Software Solutions | Pune
>>>>>>> "Don't Complain about Heat, Plant a tree"
>>>>>>>
>>>>>>> On Tue, Jun 5, 2018 at 10:15 AM, Aditya Toshniwal <
>>>>>>> [email protected]> wrote:
>>>>>>>
>>>>>>>> Hi
>>>>>>>>
>>>>>>>> On Tue, Jun 5, 2018 at 1:08 AM, Joao De Almeida Pereira <
>>>>>>>> [email protected]> wrote:
>>>>>>>>
>>>>>>>>> Hello Aditya,
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> There is no change related to notifications in this patch.
>>>>>>>>>> The below code is minor fix related to connection status of sql
>>>>>>>>>> editor. Can you please share the code snippet if it is not the below.
>>>>>>>>>>
>>>>>>>>>> - # Check for the asynchronous notifies statements.
>>>>>>>>>> - conn.check_notifies(True)
>>>>>>>>>> - notifies = conn.get_notifies()
>>>>>>>>>> + if status is not None:
>>>>>>>>>> + # Check for the asynchronous notifies statements.
>>>>>>>>>> + conn.check_notifies(True)
>>>>>>>>>> + notifies = conn.get_notifies()
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> This is a minor fix, but is it related to querying SQL_ASCII
>>>>>>>>> database?
>>>>>>>>>
>>>>>>>> No its not. It is something I found when I was working on SQL_ASCII
>>>>>>>> related changes.
>>>>>>>> Well then, will send a separate patch for it.
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Thanks
>>>>>>>>> Victoria && Joao
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Dave Page
>>>>>> Blog: http://pgsnake.blogspot.com
>>>>>> Twitter: @pgsnake
>>>>>>
>>>>>> EnterpriseDB UK: http://www.enterprisedb.com
>>>>>> The Enterprise PostgreSQL Company
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Dave Page
>>>> Blog: http://pgsnake.blogspot.com
>>>> Twitter: @pgsnake
>>>>
>>>> EnterpriseDB UK: http://www.enterprisedb.com
>>>> The Enterprise PostgreSQL Company
>>>>
>>>
>>>
>>
>>
>> --
>> Dave Page
>> Blog: http://pgsnake.blogspot.com
>> Twitter: @pgsnake
>>
>> EnterpriseDB UK: http://www.enterprisedb.com
>> The Enterprise PostgreSQL Company
>>
>
>
--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake
EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 38+ messages in thread
* Re: [pgAdmin4][RM#3289] Can't query SQL_ASCII database.
@ 2018-06-06 13:02 Aditya Toshniwal <[email protected]>
parent: Dave Page <[email protected]>
0 siblings, 1 reply; 38+ messages in thread
From: Aditya Toshniwal @ 2018-06-06 13:02 UTC (permalink / raw)
To: Dave Page <[email protected]>; +Cc: Joao De Almeida Pereira <[email protected]>; pgadmin-hackers
Hi Hackers,
PFA updated patch as the previous one was not working as expected. I have
tried to make it similar to that of pgAdmin3 and you do not need to change
client_encoding as it is set now based on server encoding. It works fine
with "view data" also.
The only problem is, I cannot find equivalent codec for wxConvLibc in
python. The closest one I could find is raw_unicode_escape. So, in a
SQL_ASCII database, non ASCII characters may differ in pgAdmin4 and
pgAdmin3, but it will display results.
Dave,
You need to add "E" before the string to be inserted, otherwise \x will be
considered as a plain string.
INSERT INTO sql_ascii (data) VALUES (E'[Invalid UTF-8] Blob:
\xf4\xa5\xa3\xa5');
Kindly review.
Thanks and Regards,
Aditya Toshniwal
Software Engineer | EnterpriseDB Software Solutions | Pune
"Don't Complain about Heat, Plant a tree"
On Tue, Jun 5, 2018 at 6:42 PM, Dave Page <[email protected]> wrote:
> Hi
>
> On Tue, Jun 5, 2018 at 2:03 PM, Aditya Toshniwal <aditya.toshniwal@
> enterprisedb.com> wrote:
>
>> Hi
>>
>> On Tue, Jun 5, 2018 at 6:25 PM, Dave Page <[email protected]> wrote:
>>
>>>
>>>
>>> On Tue, Jun 5, 2018 at 1:49 PM, Aditya Toshniwal <
>>> [email protected]> wrote:
>>>
>>>> Hi Dave,
>>>>
>>>> The problem of SQL ASCII is solved with the patch, and not related to
>>>> setting the client encoding of the sql window.
>>>>
>>>
>>> No it's not. It doesn't work for me as I said (and showed the example
>>> of).
>>>
>>
>> After setting the client_encoding to SQL_ASCII you got the output.
>> Previously, it used to fail in the back end itself because python encoding
>> failure. That is fixed.
>> The error ERROR: invalid byte sequence for encoding "UTF8": 0x80 is
>> thrown by postgres and not python or pgAdmin4. You will get the same error
>> even if you
>> connect from psql.
>>
>
> Sure - but that is not a fix. You have no way of running the SET command
> if you're using "view data" - and in the query tool, users just expect it
> to work (as it did in pgAdmin 3).
>
>
>>
>>>
>>>> I can see there is no SET call in pgAdmin3 for client_encoding. I can
>>>> remove the SET client_encoding='UNICODE'; that will solve the problem.
>>>> But, can you please let me know why that was added.
>>>>
>>>
>>> There is, but it's inside an API call (PQsetClientEncoding):
>>>
>>> 300
>>> <https://git.postgresql.org/gitweb/?p=pgadmin3.git;a=blob;f=pgadmin/db/pgConn.cpp#l300;
>>> wxLogInfo(wxT("Setting client_encoding to '%s'")
>>> , encoding.c_str());
>>> 301
>>> <https://git.postgresql.org/gitweb/?p=pgadmin3.git;a=blob;f=pgadmin/db/pgConn.cpp#l301;
>>> if (PQsetClientEncoding(conn, encoding.ToAscii()))
>>> 302
>>> <https://git.postgresql.org/gitweb/?p=pgadmin3.git;a=blob;f=pgadmin/db/pgConn.cpp#l302;
>>> {
>>> 303
>>> <https://git.postgresql.org/gitweb/?p=pgadmin3.git;a=blob;f=pgadmin/db/pgConn.cpp#l303;
>>> wxLogError(wxT("%s"), GetLastError().c_str());
>>> 304
>>> <https://git.postgresql.org/gitweb/?p=pgadmin3.git;a=blob;f=pgadmin/db/pgConn.cpp#l304;
>>> }
>>> 305
>>> <https://git.postgresql.org/gitweb/?p=pgadmin3.git;a=blob;f=pgadmin/db/pgConn.cpp#l305;
>>>
>>> Oops ! Missed that. Apologies.
>>
>>>
>>>
>>>>
>>>> Will remove the set call and will send you the updated patch if
>>>> everything works fine.
>>>>
>>>
>>> No, we need to ensure the client encoding is set correctly. It just
>>> needs to be set to SQL_ASCII if it's a SQL_ASCII database (I believe).
>>>
>>>
>> Need to rework on the initialise method. Will come with an updated.
>> patch. Sorry for trouble.
>>
>>>
>>>>
>>>> On Tue, Jun 5, 2018 at 6:05 PM, Dave Page <[email protected]> wrote:
>>>>
>>>>> Hi
>>>>>
>>>>> On Tue, Jun 5, 2018 at 1:21 PM, Aditya Toshniwal <
>>>>> [email protected]> wrote:
>>>>>
>>>>>> Hi Dave,
>>>>>>
>>>>>>
>>>>>> On Tue, Jun 5, 2018 at 4:56 PM, Dave Page <[email protected]> wrote:
>>>>>>
>>>>>>> Hi
>>>>>>>
>>>>>>> On Tue, Jun 5, 2018 at 9:50 AM, Aditya Toshniwal <
>>>>>>> [email protected]> wrote:
>>>>>>>
>>>>>>>> Hi Hackers,
>>>>>>>>
>>>>>>>> PFA updated patch. The sqleditor change is sent separately and
>>>>>>>> removed from current patch as suggested.
>>>>>>>> The test cases were running fine when the module was specified
>>>>>>>> using --pkg but were failing in complete run. Fixed that.
>>>>>>>>
>>>>>>>
>>>>>>> I did a quick test by creating a SQL_ASCII database containing a
>>>>>>> simple table:
>>>>>>>
>>>>>>> CREATE TABLE sql_ascii (id serial primary key, data text);
>>>>>>>
>>>>>>> And then populated it with data:
>>>>>>>
>>>>>>> /Library/PostgreSQL/9.4/bin/psql -d sql_ascii -U postgres -c
>>>>>>> "INSERT INTO sql_acsii (data) VALUES ('[Windows-1252] Euro: \x80 Double
>>>>>>> dagger: \x87');"
>>>>>>> /Library/PostgreSQL/9.4/bin/psql -d sql_ascii -U postgres -c
>>>>>>> "INSERT INTO sql_ascii (data) VALUES ('[Latin-1] Yen: \xa5 Half:
>>>>>>> \xbd');"
>>>>>>> /Library/PostgreSQL/9.4/bin/psql -d sql_ascii -U postgres -c
>>>>>>> "INSERT INTO sql_ascii (data) VALUES ('[Japanese] Ship: \xe8\x88\xb9');"
>>>>>>> /Library/PostgreSQL/9.4/bin/psql -d sql_ascii -U postgres -c
>>>>>>> "INSERT INTO sql_ascii (data) VALUES ('[Invalid UTF-8] Blob:
>>>>>>> \xf4\xa5\xa3\xa5');"
>>>>>>>
>>>>>>> I then right-clicked the table in the treeview, and selected the
>>>>>>> option to view all rows, and immediately saw an error:
>>>>>>>
>>>>>>> 2018-06-05 12:23:27,319: SQL pgadmin: Execute (async) for server #1
>>>>>>> - CONN:1187535 (Query-id: 8522474):
>>>>>>> SELECT * FROM public.sql_ascii
>>>>>>> ORDER BY id ASC
>>>>>>> 2018-06-05 12:23:27,320: ERROR pgadmin: Failed to execute query
>>>>>>> (execute_async) for the server #1 - CONN:1187535(Query-id: 8522474):
>>>>>>> Error Message:ERROR: invalid byte sequence for encoding "UTF8": 0x80
>>>>>>> SQL state: 22021
>>>>>>>
>>>>>>> Running "SELECT * FROM sql_ascii" in the query tool resulted in the
>>>>>>> same error, however, if I ran "SET client_encoding = 'SQL_ASCII';" first, I
>>>>>>> do see results.
>>>>>>>
>>>>>>> I have confirmed that I've restarted the server after applying the
>>>>>>> patch.
>>>>>>>
>>>>>>> What am I missing? Why don't we just set the client_encoding to
>>>>>>> SQL_ASCII if it's a SQL_ASCII database?
>>>>>>>
>>>>>>
>>>>>> It is by default same as the server encoding. But, the following
>>>>>> existing code in web/pgadmin/utils/driver/psycopg2/connection.py makes
>>>>>> the client_encoding as UNICODE for every connection. I am not sure it
>>>>>> should be removed.
>>>>>>
>>>>>> status = _execute(cur, "SET DateStyle=ISO;"
>>>>>>
>>>>>> "SET client_min_messages=notice;"
>>>>>>
>>>>>> "SET bytea_output=escape;"
>>>>>>
>>>>>> "SET client_encoding='UNICODE';")
>>>>>>
>>>>>
>>>>> It was probably before you joined, but I have said a number of times
>>>>> that pgAdmin 3 handled this differently and that maybe we should do it the
>>>>> same way here. See https://git.postgresql.org
>>>>> /gitweb/?p=pgadmin3.git;a=blob;f=pgadmin/db/pgConn.cpp, in the
>>>>> pgConn::Initialize() function.
>>>>>
>>>>> Either way, your patch isn't working for me.
>>>>>
>>>>>
>>>>>>
>>>>>>
>>>>>> Note that this testing was on Python 2.7.10 on MacOS.
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> Kindly review.
>>>>>>>>
>>>>>>>> Thanks and Regards,
>>>>>>>> Aditya Toshniwal
>>>>>>>> Software Engineer | EnterpriseDB Software Solutions | Pune
>>>>>>>> "Don't Complain about Heat, Plant a tree"
>>>>>>>>
>>>>>>>> On Tue, Jun 5, 2018 at 10:15 AM, Aditya Toshniwal <
>>>>>>>> [email protected]> wrote:
>>>>>>>>
>>>>>>>>> Hi
>>>>>>>>>
>>>>>>>>> On Tue, Jun 5, 2018 at 1:08 AM, Joao De Almeida Pereira <
>>>>>>>>> [email protected]> wrote:
>>>>>>>>>
>>>>>>>>>> Hello Aditya,
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> There is no change related to notifications in this patch.
>>>>>>>>>>> The below code is minor fix related to connection status of sql
>>>>>>>>>>> editor. Can you please share the code snippet if it is not the below.
>>>>>>>>>>>
>>>>>>>>>>> - # Check for the asynchronous notifies statements.
>>>>>>>>>>> - conn.check_notifies(True)
>>>>>>>>>>> - notifies = conn.get_notifies()
>>>>>>>>>>> + if status is not None:
>>>>>>>>>>> + # Check for the asynchronous notifies statements.
>>>>>>>>>>> + conn.check_notifies(True)
>>>>>>>>>>> + notifies = conn.get_notifies()
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> This is a minor fix, but is it related to querying SQL_ASCII
>>>>>>>>>> database?
>>>>>>>>>>
>>>>>>>>> No its not. It is something I found when I was working on
>>>>>>>>> SQL_ASCII related changes.
>>>>>>>>> Well then, will send a separate patch for it.
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Thanks
>>>>>>>>>> Victoria && Joao
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Dave Page
>>>>>>> Blog: http://pgsnake.blogspot.com
>>>>>>> Twitter: @pgsnake
>>>>>>>
>>>>>>> EnterpriseDB UK: http://www.enterprisedb.com
>>>>>>> The Enterprise PostgreSQL Company
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Dave Page
>>>>> Blog: http://pgsnake.blogspot.com
>>>>> Twitter: @pgsnake
>>>>>
>>>>> EnterpriseDB UK: http://www.enterprisedb.com
>>>>> The Enterprise PostgreSQL Company
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Dave Page
>>> Blog: http://pgsnake.blogspot.com
>>> Twitter: @pgsnake
>>>
>>> EnterpriseDB UK: http://www.enterprisedb.com
>>> The Enterprise PostgreSQL Company
>>>
>>
>>
>
>
> --
> Dave Page
> Blog: http://pgsnake.blogspot.com
> Twitter: @pgsnake
>
> EnterpriseDB UK: http://www.enterprisedb.com
> The Enterprise PostgreSQL Company
>
Attachments:
[text/x-patch] RM3289.patch (9.7K, 3-RM3289.patch)
download | inline diff:
diff --git a/web/pgadmin/tools/sqleditor/tests/test_encoding_charset.py b/web/pgadmin/tools/sqleditor/tests/test_encoding_charset.py
new file mode 100644
index 0000000..0826dec
--- /dev/null
+++ b/web/pgadmin/tools/sqleditor/tests/test_encoding_charset.py
@@ -0,0 +1,103 @@
+##########################################################################
+#
+# pgAdmin 4 - PostgreSQL Tools
+#
+# Copyright (C) 2013 - 2018, The pgAdmin Development Team
+# This software is released under the PostgreSQL Licence
+#
+##########################################################################
+
+from pgadmin.utils.route import BaseTestGenerator
+from pgadmin.browser.server_groups.servers.databases.tests import utils as \
+ database_utils
+from regression import parent_node_dict
+from regression.python_test_utils import test_utils
+import json
+
+
+class TestEncodingCharset(BaseTestGenerator):
+ """
+ This class validates character support in pgAdmin4 for
+ different PostgresDB encodings
+ """
+ scenarios = [
+ (
+ 'With Encoding UTF8',
+ dict(
+ db_encoding='UTF8',
+ lc_collate='C',
+ test_str='A'
+ )),
+ (
+ 'With Encoding WIN1252',
+ dict(
+ db_encoding='WIN1252',
+ lc_collate='C',
+ test_str='A'
+ )),
+ (
+ 'With Encoding EUC_CN',
+ dict(
+ db_encoding='EUC_CN',
+ lc_collate='C',
+ test_str='A'
+ )),
+ (
+ 'With Encoding SQL_ASCII',
+ dict(
+ db_encoding='SQL_ASCII',
+ lc_collate='C',
+ test_str='\\255'
+ )),
+ ]
+
+ def setUp(self):
+ self.encode_db_name = 'encoding_' + self.db_encoding
+ self.encode_sid = self.server_information['server_id']
+ self.encode_did = test_utils.create_database(
+ self.server, self.encode_db_name,
+ (self.db_encoding, self.lc_collate))
+
+ def runTest(self):
+
+ db_con = database_utils.connect_database(self,
+ test_utils.SERVER_GROUP,
+ self.encode_sid,
+ self.encode_did)
+ if not db_con["info"] == "Database connected.":
+ raise Exception("Could not connect to the database.")
+
+ # Initialize query tool
+ url = '/datagrid/initialize/query_tool/{0}/{1}/{2}'.format(
+ test_utils.SERVER_GROUP, self.encode_sid, self.encode_did)
+ response = self.tester.post(url)
+ self.assertEquals(response.status_code, 200)
+
+ response_data = json.loads(response.data.decode('utf-8'))
+ self.trans_id = response_data['data']['gridTransId']
+
+ # Check character
+ url = "/sqleditor/query_tool/start/{0}".format(self.trans_id)
+ sql = "select E'{0}';".format(self.test_str)
+ response = self.tester.post(url, data=json.dumps({"sql": sql}),
+ content_type='html/json')
+ self.assertEquals(response.status_code, 200)
+ url = '/sqleditor/poll/{0}'.format(self.trans_id)
+ response = self.tester.get(url)
+ self.assertEquals(response.status_code, 200)
+ response_data = json.loads(response.data.decode('utf-8'))
+ self.assertEquals(response_data['data']['rows_fetched_to'], 1)
+
+ database_utils.disconnect_database(self, self.encode_sid,
+ self.encode_did)
+
+ def tearDown(self):
+ main_conn = test_utils.get_db_connection(
+ self.server['db'],
+ self.server['username'],
+ self.server['db_password'],
+ self.server['host'],
+ self.server['port'],
+ self.server['sslmode']
+ )
+ test_utils.drop_database(main_conn, self.encode_db_name)
diff --git a/web/pgadmin/utils/driver/psycopg2/connection.py b/web/pgadmin/utils/driver/psycopg2/connection.py
index cfd161a..e2538d1 100644
--- a/web/pgadmin/utils/driver/psycopg2/connection.py
+++ b/web/pgadmin/utils/driver/psycopg2/connection.py
@@ -33,7 +33,7 @@ from pgadmin.utils import get_complete_file_path
from ..abstract import BaseDriver, BaseConnection
from .cursor import DictCursor
from .typecast import register_global_typecasters, \
- register_string_typecasters, register_binary_typecasters, \
+ register_binary_typecasters, \
register_array_to_string_typecasters, ALL_JSON_TYPES
@@ -387,8 +387,6 @@ class Connection(BaseConnection):
else:
self.conn.autocommit = True
- register_string_typecasters(self.conn)
-
if self.array_to_string:
register_array_to_string_typecasters(self.conn)
@@ -397,10 +395,19 @@ class Connection(BaseConnection):
if self.use_binary_placeholder:
register_binary_typecasters(self.conn)
- status = _execute(cur, "SET DateStyle=ISO;"
- "SET client_min_messages=notice;"
- "SET bytea_output=escape;"
- "SET client_encoding='UNICODE';")
+ if self.conn.encoding not in ('SQL_ASCII', 'SQLASCII',
+ 'MULE_INTERNAL', 'MULEINTERNAL'):
+ status = _execute(cur, "SET DateStyle=ISO;"
+ "SET client_min_messages=notice;"
+ "SET bytea_output=escape;"
+ "SET client_encoding='UNICODE';")
+
+ encodings[self.conn.encoding] = 'utf-8'
+ else:
+ status = _execute(cur, "SET DateStyle=ISO;"
+ "SET client_min_messages=notice;"
+ "SET bytea_output=escape;")
+ encodings[self.conn.encoding] = 'raw_unicode_escape'
if status is not None:
self.conn.close()
diff --git a/web/pgadmin/utils/driver/psycopg2/typecast.py b/web/pgadmin/utils/driver/psycopg2/typecast.py
index f136604..a8f6c38 100644
--- a/web/pgadmin/utils/driver/psycopg2/typecast.py
+++ b/web/pgadmin/utils/driver/psycopg2/typecast.py
@@ -163,49 +163,6 @@ def register_global_typecasters():
psycopg2.extensions.register_type(pg_array_types_to_array_of_string_type)
-def register_string_typecasters(connection):
- if connection.encoding != 'UTF8':
- # In python3 when database encoding is other than utf-8 and client
- # encoding is set to UNICODE then we need to map data from database
- # encoding to utf-8.
- # This is required because when client encoding is set to UNICODE then
- # psycopg assumes database encoding utf-8 and not the actual encoding.
- # Not sure whether it's bug or feature in psycopg for python3.
- if sys.version_info >= (3,):
- def return_as_unicode(value, cursor):
- if value is None:
- return None
- # Treat value as byte sequence of database encoding and then
- # decode it as utf-8 to get correct unicode value.
- return bytes(
- value, encodings[cursor.connection.encoding]
- ).decode('utf-8')
-
- unicode_type = psycopg2.extensions.new_type(
- # "char", name, text, character, character varying
- (19, 18, 25, 1042, 1043, 0),
- 'UNICODE', return_as_unicode)
- else:
- def return_as_unicode(value, cursor):
- if value is None:
- return None
- # Decode it as utf-8 to get correct unicode value.
- return value.decode('utf-8')
-
- unicode_type = psycopg2.extensions.new_type(
- # "char", name, text, character, character varying
- (19, 18, 25, 1042, 1043, 0),
- 'UNICODE', return_as_unicode)
-
- unicode_array_type = psycopg2.extensions.new_array_type(
- # "char"[], name[], text[], character[], character varying[]
- (1002, 1003, 1009, 1014, 1015, 0
- ), 'UNICODEARRAY', unicode_type)
-
- psycopg2.extensions.register_type(unicode_type)
- psycopg2.extensions.register_type(unicode_array_type)
-
-
def register_binary_typecasters(connection):
psycopg2.extensions.register_type(
psycopg2.extensions.new_type(
diff --git a/web/regression/python_test_utils/test_utils.py b/web/regression/python_test_utils/test_utils.py
index 3e517b6..464a09e 100644
--- a/web/regression/python_test_utils/test_utils.py
+++ b/web/regression/python_test_utils/test_utils.py
@@ -116,7 +116,7 @@ def clear_node_info_dict():
del node_info_dict[node][:]
-def create_database(server, db_name):
+def create_database(server, db_name, encoding=None):
"""This function used to create database and returns the database id"""
try:
connection = get_db_connection(
@@ -130,8 +130,14 @@ def create_database(server, db_name):
old_isolation_level = connection.isolation_level
connection.set_isolation_level(0)
pg_cursor = connection.cursor()
- pg_cursor.execute(
- '''CREATE DATABASE "%s" TEMPLATE template0''' % db_name)
+ if encoding is None:
+ pg_cursor.execute(
+ '''CREATE DATABASE "%s" TEMPLATE template0''' % db_name)
+ else:
+ pg_cursor.execute(
+ '''CREATE DATABASE "%s" TEMPLATE template0
+ ENCODING='%s' LC_COLLATE='%s' LC_CTYPE='%s' ''' %
+ (db_name, encoding[0], encoding[1], encoding[1]))
connection.set_isolation_level(old_isolation_level)
connection.commit()
^ permalink raw reply [nested|flat] 38+ messages in thread
* Re: [pgAdmin4][RM#3289] Can't query SQL_ASCII database.
@ 2018-06-07 10:37 Dave Page <[email protected]>
parent: Aditya Toshniwal <[email protected]>
0 siblings, 1 reply; 38+ messages in thread
From: Dave Page @ 2018-06-07 10:37 UTC (permalink / raw)
To: Aditya Toshniwal <[email protected]>; +Cc: Joao De Almeida Pereira <[email protected]>; pgadmin-hackers
Hi
On Wed, Jun 6, 2018 at 2:02 PM, Aditya Toshniwal <aditya.toshniwal@
enterprisedb.com> wrote:
> Hi Hackers,
>
> PFA updated patch as the previous one was not working as expected. I have
> tried to make it similar to that of pgAdmin3 and you do not need to change
> client_encoding as it is set now based on server encoding. It works fine
> with "view data" also.
>
- In connection.py, at ~409, shouldn't we set the client_encoding to
SQL_ASCII? Otherwise it could be overridden with something unexpected if
the client has PGCLIENTENCODING set for example.
- With or without that change, I get the following test failure on macOS
with Python 2.7.10:
======================================================================
ERROR: runTest
(pgadmin.tools.sqleditor.tests.test_encoding_charset.TestEncodingCharset)
With Encoding SQL_ASCII
----------------------------------------------------------------------
Traceback (most recent call last):
File
"/Users/dpage/git/pgadmin4/web/pgadmin/tools/sqleditor/tests/test_encoding_charset.py",
line 86, in runTest
response = self.tester.get(url)
File
"/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/werkzeug/test.py",
line 830, in get
return self.open(*args, **kw)
File
"/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/testing.py",
line 127, in open
follow_redirects=follow_redirects)
File
"/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/werkzeug/test.py",
line 803, in open
response = self.run_wsgi_app(environ, buffered=buffered)
File
"/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/werkzeug/test.py",
line 716, in run_wsgi_app
rv = run_wsgi_app(self.application, environ, buffered=buffered)
File
"/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/werkzeug/test.py",
line 923, in run_wsgi_app
app_rv = app(environ, start_response)
File
"/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
line 1997, in __call__
return self.wsgi_app(environ, start_response)
File
"/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
line 1985, in wsgi_app
response = self.handle_exception(e)
File
"/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
line 1540, in handle_exception
reraise(exc_type, exc_value, tb)
File
"/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
line 1982, in wsgi_app
response = self.full_dispatch_request()
File
"/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
line 1614, in full_dispatch_request
rv = self.handle_user_exception(e)
File
"/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
line 1517, in handle_user_exception
reraise(exc_type, exc_value, tb)
File
"/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
line 1612, in full_dispatch_request
rv = self.dispatch_request()
File
"/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
line 1598, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File
"/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask_login.py",
line 792, in decorated_view
return func(*args, **kwargs)
File "/Users/dpage/git/pgadmin4/web/pgadmin/tools/sqleditor/__init__.py",
line 576, in poll
'oids': oids
File "/Users/dpage/git/pgadmin4/web/pgadmin/utils/ajax.py", line 61, in
make_json_response
separators=(',', ':')),
File
"/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/simplejson/__init__.py",
line 399, in dumps
**kw).encode(obj)
File
"/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/simplejson/encoder.py",
line 291, in encode
chunks = self.iterencode(o, _one_shot=True)
File
"/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/simplejson/encoder.py",
line 373, in iterencode
return _iterencode(o, 0)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xad in position 0:
invalid start byte
----------------------------------------------------------------------
Ran 317 tests in 30.692s
FAILED (errors=1, skipped=21)
> The only problem is, I cannot find equivalent codec for wxConvLibc in
> python. The closest one I could find is raw_unicode_escape. So, in a
> SQL_ASCII database, non ASCII characters may differ in pgAdmin4 and
> pgAdmin3, but it will display results.
>
Yeah, I think that's fine. For the small number of people with SQL_ASCII
databases, seeing escaped characters is better than nothing.
>
>
> Dave,
> You need to add "E" before the string to be inserted, otherwise \x will be
> considered as a plain string.
> INSERT INTO sql_ascii (data) VALUES (E'[Invalid UTF-8] Blob:
> \xf4\xa5\xa3\xa5');
>
Yeah, sorry - I copied the wrong version of the query :-(
>
>
> Kindly review.
>
> Thanks and Regards,
> Aditya Toshniwal
> Software Engineer | EnterpriseDB Software Solutions | Pune
> "Don't Complain about Heat, Plant a tree"
>
> On Tue, Jun 5, 2018 at 6:42 PM, Dave Page <[email protected]> wrote:
>
>> Hi
>>
>> On Tue, Jun 5, 2018 at 2:03 PM, Aditya Toshniwal <
>> [email protected]> wrote:
>>
>>> Hi
>>>
>>> On Tue, Jun 5, 2018 at 6:25 PM, Dave Page <[email protected]> wrote:
>>>
>>>>
>>>>
>>>> On Tue, Jun 5, 2018 at 1:49 PM, Aditya Toshniwal <
>>>> [email protected]> wrote:
>>>>
>>>>> Hi Dave,
>>>>>
>>>>> The problem of SQL ASCII is solved with the patch, and not related to
>>>>> setting the client encoding of the sql window.
>>>>>
>>>>
>>>> No it's not. It doesn't work for me as I said (and showed the example
>>>> of).
>>>>
>>>
>>> After setting the client_encoding to SQL_ASCII you got the output.
>>> Previously, it used to fail in the back end itself because python encoding
>>> failure. That is fixed.
>>> The error ERROR: invalid byte sequence for encoding "UTF8": 0x80 is
>>> thrown by postgres and not python or pgAdmin4. You will get the same error
>>> even if you
>>> connect from psql.
>>>
>>
>> Sure - but that is not a fix. You have no way of running the SET command
>> if you're using "view data" - and in the query tool, users just expect it
>> to work (as it did in pgAdmin 3).
>>
>>
>>>
>>>>
>>>>> I can see there is no SET call in pgAdmin3 for client_encoding. I can
>>>>> remove the SET client_encoding='UNICODE'; that will solve the
>>>>> problem. But, can you please let me know why that was added.
>>>>>
>>>>
>>>> There is, but it's inside an API call (PQsetClientEncoding):
>>>>
>>>> 300
>>>> <https://git.postgresql.org/gitweb/?p=pgadmin3.git;a=blob;f=pgadmin/db/pgConn.cpp#l300;
>>>> wxLogInfo(wxT("Setting client_encoding to '%s'")
>>>> , encoding.c_str());
>>>> 301
>>>> <https://git.postgresql.org/gitweb/?p=pgadmin3.git;a=blob;f=pgadmin/db/pgConn.cpp#l301;
>>>> if (PQsetClientEncoding(conn, encoding.ToAscii()))
>>>> 302
>>>> <https://git.postgresql.org/gitweb/?p=pgadmin3.git;a=blob;f=pgadmin/db/pgConn.cpp#l302;
>>>> {
>>>> 303
>>>> <https://git.postgresql.org/gitweb/?p=pgadmin3.git;a=blob;f=pgadmin/db/pgConn.cpp#l303;
>>>> wxLogError(wxT("%s"), GetLastError().c_str());
>>>> 304
>>>> <https://git.postgresql.org/gitweb/?p=pgadmin3.git;a=blob;f=pgadmin/db/pgConn.cpp#l304;
>>>> }
>>>> 305
>>>> <https://git.postgresql.org/gitweb/?p=pgadmin3.git;a=blob;f=pgadmin/db/pgConn.cpp#l305;
>>>>
>>>> Oops ! Missed that. Apologies.
>>>
>>>>
>>>>
>>>>>
>>>>> Will remove the set call and will send you the updated patch if
>>>>> everything works fine.
>>>>>
>>>>
>>>> No, we need to ensure the client encoding is set correctly. It just
>>>> needs to be set to SQL_ASCII if it's a SQL_ASCII database (I believe).
>>>>
>>>>
>>> Need to rework on the initialise method. Will come with an updated.
>>> patch. Sorry for trouble.
>>>
>>>>
>>>>>
>>>>> On Tue, Jun 5, 2018 at 6:05 PM, Dave Page <[email protected]> wrote:
>>>>>
>>>>>> Hi
>>>>>>
>>>>>> On Tue, Jun 5, 2018 at 1:21 PM, Aditya Toshniwal <
>>>>>> [email protected]> wrote:
>>>>>>
>>>>>>> Hi Dave,
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Jun 5, 2018 at 4:56 PM, Dave Page <[email protected]> wrote:
>>>>>>>
>>>>>>>> Hi
>>>>>>>>
>>>>>>>> On Tue, Jun 5, 2018 at 9:50 AM, Aditya Toshniwal <
>>>>>>>> [email protected]> wrote:
>>>>>>>>
>>>>>>>>> Hi Hackers,
>>>>>>>>>
>>>>>>>>> PFA updated patch. The sqleditor change is sent separately and
>>>>>>>>> removed from current patch as suggested.
>>>>>>>>> The test cases were running fine when the module was specified
>>>>>>>>> using --pkg but were failing in complete run. Fixed that.
>>>>>>>>>
>>>>>>>>
>>>>>>>> I did a quick test by creating a SQL_ASCII database containing a
>>>>>>>> simple table:
>>>>>>>>
>>>>>>>> CREATE TABLE sql_ascii (id serial primary key, data text);
>>>>>>>>
>>>>>>>> And then populated it with data:
>>>>>>>>
>>>>>>>> /Library/PostgreSQL/9.4/bin/psql -d sql_ascii -U postgres -c
>>>>>>>> "INSERT INTO sql_acsii (data) VALUES ('[Windows-1252] Euro: \x80 Double
>>>>>>>> dagger: \x87');"
>>>>>>>> /Library/PostgreSQL/9.4/bin/psql -d sql_ascii -U postgres -c
>>>>>>>> "INSERT INTO sql_ascii (data) VALUES ('[Latin-1] Yen: \xa5 Half:
>>>>>>>> \xbd');"
>>>>>>>> /Library/PostgreSQL/9.4/bin/psql -d sql_ascii -U postgres -c
>>>>>>>> "INSERT INTO sql_ascii (data) VALUES ('[Japanese] Ship: \xe8\x88\xb9');"
>>>>>>>> /Library/PostgreSQL/9.4/bin/psql -d sql_ascii -U postgres -c
>>>>>>>> "INSERT INTO sql_ascii (data) VALUES ('[Invalid UTF-8] Blob:
>>>>>>>> \xf4\xa5\xa3\xa5');"
>>>>>>>>
>>>>>>>> I then right-clicked the table in the treeview, and selected the
>>>>>>>> option to view all rows, and immediately saw an error:
>>>>>>>>
>>>>>>>> 2018-06-05 12:23:27,319: SQL pgadmin: Execute (async) for server
>>>>>>>> #1 - CONN:1187535 (Query-id: 8522474):
>>>>>>>> SELECT * FROM public.sql_ascii
>>>>>>>> ORDER BY id ASC
>>>>>>>> 2018-06-05 12:23:27,320: ERROR pgadmin: Failed to execute query
>>>>>>>> (execute_async) for the server #1 - CONN:1187535(Query-id: 8522474):
>>>>>>>> Error Message:ERROR: invalid byte sequence for encoding "UTF8":
>>>>>>>> 0x80
>>>>>>>> SQL state: 22021
>>>>>>>>
>>>>>>>> Running "SELECT * FROM sql_ascii" in the query tool resulted in the
>>>>>>>> same error, however, if I ran "SET client_encoding = 'SQL_ASCII';" first, I
>>>>>>>> do see results.
>>>>>>>>
>>>>>>>> I have confirmed that I've restarted the server after applying the
>>>>>>>> patch.
>>>>>>>>
>>>>>>>> What am I missing? Why don't we just set the client_encoding to
>>>>>>>> SQL_ASCII if it's a SQL_ASCII database?
>>>>>>>>
>>>>>>>
>>>>>>> It is by default same as the server encoding. But, the following
>>>>>>> existing code in web/pgadmin/utils/driver/psycopg2/connection.py makes
>>>>>>> the client_encoding as UNICODE for every connection. I am not sure it
>>>>>>> should be removed.
>>>>>>>
>>>>>>> status = _execute(cur, "SET DateStyle=ISO;"
>>>>>>>
>>>>>>> "SET client_min_messages=notice;"
>>>>>>>
>>>>>>> "SET bytea_output=escape;"
>>>>>>>
>>>>>>> "SET client_encoding='UNICODE';")
>>>>>>>
>>>>>>
>>>>>> It was probably before you joined, but I have said a number of times
>>>>>> that pgAdmin 3 handled this differently and that maybe we should do it the
>>>>>> same way here. See https://git.postgresql.org
>>>>>> /gitweb/?p=pgadmin3.git;a=blob;f=pgadmin/db/pgConn.cpp, in the
>>>>>> pgConn::Initialize() function.
>>>>>>
>>>>>> Either way, your patch isn't working for me.
>>>>>>
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Note that this testing was on Python 2.7.10 on MacOS.
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Kindly review.
>>>>>>>>>
>>>>>>>>> Thanks and Regards,
>>>>>>>>> Aditya Toshniwal
>>>>>>>>> Software Engineer | EnterpriseDB Software Solutions | Pune
>>>>>>>>> "Don't Complain about Heat, Plant a tree"
>>>>>>>>>
>>>>>>>>> On Tue, Jun 5, 2018 at 10:15 AM, Aditya Toshniwal <
>>>>>>>>> [email protected]> wrote:
>>>>>>>>>
>>>>>>>>>> Hi
>>>>>>>>>>
>>>>>>>>>> On Tue, Jun 5, 2018 at 1:08 AM, Joao De Almeida Pereira <
>>>>>>>>>> [email protected]> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hello Aditya,
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> There is no change related to notifications in this patch.
>>>>>>>>>>>> The below code is minor fix related to connection status of sql
>>>>>>>>>>>> editor. Can you please share the code snippet if it is not the below.
>>>>>>>>>>>>
>>>>>>>>>>>> - # Check for the asynchronous notifies statements.
>>>>>>>>>>>> - conn.check_notifies(True)
>>>>>>>>>>>> - notifies = conn.get_notifies()
>>>>>>>>>>>> + if status is not None:
>>>>>>>>>>>> + # Check for the asynchronous notifies statements.
>>>>>>>>>>>> + conn.check_notifies(True)
>>>>>>>>>>>> + notifies = conn.get_notifies()
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>> This is a minor fix, but is it related to querying SQL_ASCII
>>>>>>>>>>> database?
>>>>>>>>>>>
>>>>>>>>>> No its not. It is something I found when I was working on
>>>>>>>>>> SQL_ASCII related changes.
>>>>>>>>>> Well then, will send a separate patch for it.
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Thanks
>>>>>>>>>>> Victoria && Joao
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Dave Page
>>>>>>>> Blog: http://pgsnake.blogspot.com
>>>>>>>> Twitter: @pgsnake
>>>>>>>>
>>>>>>>> EnterpriseDB UK: http://www.enterprisedb.com
>>>>>>>> The Enterprise PostgreSQL Company
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Dave Page
>>>>>> Blog: http://pgsnake.blogspot.com
>>>>>> Twitter: @pgsnake
>>>>>>
>>>>>> EnterpriseDB UK: http://www.enterprisedb.com
>>>>>> The Enterprise PostgreSQL Company
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Dave Page
>>>> Blog: http://pgsnake.blogspot.com
>>>> Twitter: @pgsnake
>>>>
>>>> EnterpriseDB UK: http://www.enterprisedb.com
>>>> The Enterprise PostgreSQL Company
>>>>
>>>
>>>
>>
>>
>> --
>> Dave Page
>> Blog: http://pgsnake.blogspot.com
>> Twitter: @pgsnake
>>
>> EnterpriseDB UK: http://www.enterprisedb.com
>> The Enterprise PostgreSQL Company
>>
>
>
--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake
EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 38+ messages in thread
* Re: [pgAdmin4][RM#3289] Can't query SQL_ASCII database.
@ 2018-06-07 11:05 Aditya Toshniwal <[email protected]>
parent: Dave Page <[email protected]>
0 siblings, 1 reply; 38+ messages in thread
From: Aditya Toshniwal @ 2018-06-07 11:05 UTC (permalink / raw)
To: Dave Page <[email protected]>; +Cc: Joao De Almeida Pereira <[email protected]>; pgadmin-hackers
Hi Dave,
On Thu, Jun 7, 2018 at 4:07 PM, Dave Page <[email protected]> wrote:
> Hi
>
> On Wed, Jun 6, 2018 at 2:02 PM, Aditya Toshniwal <
> [email protected]> wrote:
>
>> Hi Hackers,
>>
>> PFA updated patch as the previous one was not working as expected. I have
>> tried to make it similar to that of pgAdmin3 and you do not need to change
>> client_encoding as it is set now based on server encoding. It works fine
>> with "view data" also.
>>
>
> - In connection.py, at ~409, shouldn't we set the client_encoding to
> SQL_ASCII? Otherwise it could be overridden with something unexpected if
> the client has PGCLIENTENCODING set for example.
>
Yeah I agree, it would be better to add. Will add the change.
>
> - With or without that change, I get the following test failure on macOS
> with Python 2.7.10:
>
It works fine on my machine with Python 2.7 and macOS. Could you please let
me know the Postgres DB version also.
Will test on few more machines.
>
> ======================================================================
> ERROR: runTest (pgadmin.tools.sqleditor.tests.test_encoding_charset.
> TestEncodingCharset)
> With Encoding SQL_ASCII
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> File "/Users/dpage/git/pgadmin4/web/pgadmin/tools/sqleditor/
> tests/test_encoding_charset.py", line 86, in runTest
> response = self.tester.get(url)
> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/werkzeug/test.py",
> line 830, in get
> return self.open(*args, **kw)
> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/testing.py",
> line 127, in open
> follow_redirects=follow_redirects)
> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/werkzeug/test.py",
> line 803, in open
> response = self.run_wsgi_app(environ, buffered=buffered)
> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/werkzeug/test.py",
> line 716, in run_wsgi_app
> rv = run_wsgi_app(self.application, environ, buffered=buffered)
> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/werkzeug/test.py",
> line 923, in run_wsgi_app
> app_rv = app(environ, start_response)
> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
> line 1997, in __call__
> return self.wsgi_app(environ, start_response)
> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
> line 1985, in wsgi_app
> response = self.handle_exception(e)
> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
> line 1540, in handle_exception
> reraise(exc_type, exc_value, tb)
> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
> line 1982, in wsgi_app
> response = self.full_dispatch_request()
> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
> line 1614, in full_dispatch_request
> rv = self.handle_user_exception(e)
> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
> line 1517, in handle_user_exception
> reraise(exc_type, exc_value, tb)
> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
> line 1612, in full_dispatch_request
> rv = self.dispatch_request()
> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
> line 1598, in dispatch_request
> return self.view_functions[rule.endpoint](**req.view_args)
> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask_login.py",
> line 792, in decorated_view
> return func(*args, **kwargs)
> File "/Users/dpage/git/pgadmin4/web/pgadmin/tools/sqleditor/__init__.py",
> line 576, in poll
> 'oids': oids
> File "/Users/dpage/git/pgadmin4/web/pgadmin/utils/ajax.py", line 61, in
> make_json_response
> separators=(',', ':')),
> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-
> packages/simplejson/__init__.py", line 399, in dumps
> **kw).encode(obj)
> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-
> packages/simplejson/encoder.py", line 291, in encode
> chunks = self.iterencode(o, _one_shot=True)
> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-
> packages/simplejson/encoder.py", line 373, in iterencode
> return _iterencode(o, 0)
> UnicodeDecodeError: 'utf8' codec can't decode byte 0xad in position 0:
> invalid start byte
>
> ----------------------------------------------------------------------
> Ran 317 tests in 30.692s
>
> FAILED (errors=1, skipped=21)
>
>
>
>> The only problem is, I cannot find equivalent codec for wxConvLibc in
>> python. The closest one I could find is raw_unicode_escape. So, in a
>> SQL_ASCII database, non ASCII characters may differ in pgAdmin4 and
>> pgAdmin3, but it will display results.
>>
>
> Yeah, I think that's fine. For the small number of people with SQL_ASCII
> databases, seeing escaped characters is better than nothing.
>
>
>>
>>
>> Dave,
>> You need to add "E" before the string to be inserted, otherwise \x will
>> be considered as a plain string.
>> INSERT INTO sql_ascii (data) VALUES (E'[Invalid UTF-8] Blob:
>> \xf4\xa5\xa3\xa5');
>>
>
> Yeah, sorry - I copied the wrong version of the query :-(
>
>
>>
>>
>> Kindly review.
>>
>> Thanks and Regards,
>> Aditya Toshniwal
>> Software Engineer | EnterpriseDB Software Solutions | Pune
>> "Don't Complain about Heat, Plant a tree"
>>
>> On Tue, Jun 5, 2018 at 6:42 PM, Dave Page <[email protected]> wrote:
>>
>>> Hi
>>>
>>> On Tue, Jun 5, 2018 at 2:03 PM, Aditya Toshniwal <
>>> [email protected]> wrote:
>>>
>>>> Hi
>>>>
>>>> On Tue, Jun 5, 2018 at 6:25 PM, Dave Page <[email protected]> wrote:
>>>>
>>>>>
>>>>>
>>>>> On Tue, Jun 5, 2018 at 1:49 PM, Aditya Toshniwal <
>>>>> [email protected]> wrote:
>>>>>
>>>>>> Hi Dave,
>>>>>>
>>>>>> The problem of SQL ASCII is solved with the patch, and not related to
>>>>>> setting the client encoding of the sql window.
>>>>>>
>>>>>
>>>>> No it's not. It doesn't work for me as I said (and showed the example
>>>>> of).
>>>>>
>>>>
>>>> After setting the client_encoding to SQL_ASCII you got the output.
>>>> Previously, it used to fail in the back end itself because python encoding
>>>> failure. That is fixed.
>>>> The error ERROR: invalid byte sequence for encoding "UTF8": 0x80 is
>>>> thrown by postgres and not python or pgAdmin4. You will get the same error
>>>> even if you
>>>> connect from psql.
>>>>
>>>
>>> Sure - but that is not a fix. You have no way of running the SET command
>>> if you're using "view data" - and in the query tool, users just expect it
>>> to work (as it did in pgAdmin 3).
>>>
>>>
>>>>
>>>>>
>>>>>> I can see there is no SET call in pgAdmin3 for client_encoding. I
>>>>>> can remove the SET client_encoding='UNICODE'; that will solve the
>>>>>> problem. But, can you please let me know why that was added.
>>>>>>
>>>>>
>>>>> There is, but it's inside an API call (PQsetClientEncoding):
>>>>>
>>>>> 300
>>>>> <https://git.postgresql.org/gitweb/?p=pgadmin3.git;a=blob;f=pgadmin/db/pgConn.cpp#l300;
>>>>> wxLogInfo(wxT("Setting client_encoding to '%s'")
>>>>> , encoding.c_str());
>>>>> 301
>>>>> <https://git.postgresql.org/gitweb/?p=pgadmin3.git;a=blob;f=pgadmin/db/pgConn.cpp#l301;
>>>>> if (PQsetClientEncoding(conn, encoding.ToAscii()))
>>>>> 302
>>>>> <https://git.postgresql.org/gitweb/?p=pgadmin3.git;a=blob;f=pgadmin/db/pgConn.cpp#l302;
>>>>> {
>>>>> 303
>>>>> <https://git.postgresql.org/gitweb/?p=pgadmin3.git;a=blob;f=pgadmin/db/pgConn.cpp#l303;
>>>>> wxLogError(wxT("%s"), GetLastError().c_str());
>>>>> 304
>>>>> <https://git.postgresql.org/gitweb/?p=pgadmin3.git;a=blob;f=pgadmin/db/pgConn.cpp#l304;
>>>>> }
>>>>> 305
>>>>> <https://git.postgresql.org/gitweb/?p=pgadmin3.git;a=blob;f=pgadmin/db/pgConn.cpp#l305;
>>>>>
>>>>> Oops ! Missed that. Apologies.
>>>>
>>>>>
>>>>>
>>>>>>
>>>>>> Will remove the set call and will send you the updated patch if
>>>>>> everything works fine.
>>>>>>
>>>>>
>>>>> No, we need to ensure the client encoding is set correctly. It just
>>>>> needs to be set to SQL_ASCII if it's a SQL_ASCII database (I believe).
>>>>>
>>>>>
>>>> Need to rework on the initialise method. Will come with an updated.
>>>> patch. Sorry for trouble.
>>>>
>>>>>
>>>>>>
>>>>>> On Tue, Jun 5, 2018 at 6:05 PM, Dave Page <[email protected]> wrote:
>>>>>>
>>>>>>> Hi
>>>>>>>
>>>>>>> On Tue, Jun 5, 2018 at 1:21 PM, Aditya Toshniwal <
>>>>>>> [email protected]> wrote:
>>>>>>>
>>>>>>>> Hi Dave,
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, Jun 5, 2018 at 4:56 PM, Dave Page <[email protected]>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi
>>>>>>>>>
>>>>>>>>> On Tue, Jun 5, 2018 at 9:50 AM, Aditya Toshniwal <
>>>>>>>>> [email protected]> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Hackers,
>>>>>>>>>>
>>>>>>>>>> PFA updated patch. The sqleditor change is sent separately and
>>>>>>>>>> removed from current patch as suggested.
>>>>>>>>>> The test cases were running fine when the module was specified
>>>>>>>>>> using --pkg but were failing in complete run. Fixed that.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I did a quick test by creating a SQL_ASCII database containing a
>>>>>>>>> simple table:
>>>>>>>>>
>>>>>>>>> CREATE TABLE sql_ascii (id serial primary key, data text);
>>>>>>>>>
>>>>>>>>> And then populated it with data:
>>>>>>>>>
>>>>>>>>> /Library/PostgreSQL/9.4/bin/psql -d sql_ascii -U postgres -c
>>>>>>>>> "INSERT INTO sql_acsii (data) VALUES ('[Windows-1252] Euro: \x80 Double
>>>>>>>>> dagger: \x87');"
>>>>>>>>> /Library/PostgreSQL/9.4/bin/psql -d sql_ascii -U postgres -c
>>>>>>>>> "INSERT INTO sql_ascii (data) VALUES ('[Latin-1] Yen: \xa5 Half:
>>>>>>>>> \xbd');"
>>>>>>>>> /Library/PostgreSQL/9.4/bin/psql -d sql_ascii -U postgres -c
>>>>>>>>> "INSERT INTO sql_ascii (data) VALUES ('[Japanese] Ship: \xe8\x88\xb9');"
>>>>>>>>> /Library/PostgreSQL/9.4/bin/psql -d sql_ascii -U postgres -c
>>>>>>>>> "INSERT INTO sql_ascii (data) VALUES ('[Invalid UTF-8] Blob:
>>>>>>>>> \xf4\xa5\xa3\xa5');"
>>>>>>>>>
>>>>>>>>> I then right-clicked the table in the treeview, and selected the
>>>>>>>>> option to view all rows, and immediately saw an error:
>>>>>>>>>
>>>>>>>>> 2018-06-05 12:23:27,319: SQL pgadmin: Execute (async) for server
>>>>>>>>> #1 - CONN:1187535 (Query-id: 8522474):
>>>>>>>>> SELECT * FROM public.sql_ascii
>>>>>>>>> ORDER BY id ASC
>>>>>>>>> 2018-06-05 12:23:27,320: ERROR pgadmin: Failed to execute query
>>>>>>>>> (execute_async) for the server #1 - CONN:1187535(Query-id: 8522474):
>>>>>>>>> Error Message:ERROR: invalid byte sequence for encoding "UTF8":
>>>>>>>>> 0x80
>>>>>>>>> SQL state: 22021
>>>>>>>>>
>>>>>>>>> Running "SELECT * FROM sql_ascii" in the query tool resulted in
>>>>>>>>> the same error, however, if I ran "SET client_encoding = 'SQL_ASCII';"
>>>>>>>>> first, I do see results.
>>>>>>>>>
>>>>>>>>> I have confirmed that I've restarted the server after applying the
>>>>>>>>> patch.
>>>>>>>>>
>>>>>>>>> What am I missing? Why don't we just set the client_encoding to
>>>>>>>>> SQL_ASCII if it's a SQL_ASCII database?
>>>>>>>>>
>>>>>>>>
>>>>>>>> It is by default same as the server encoding. But, the following
>>>>>>>> existing code in web/pgadmin/utils/driver/psycopg2/connection.py makes
>>>>>>>> the client_encoding as UNICODE for every connection. I am not sure it
>>>>>>>> should be removed.
>>>>>>>>
>>>>>>>> status = _execute(cur, "SET DateStyle=ISO;"
>>>>>>>>
>>>>>>>> "SET client_min_messages=notice;"
>>>>>>>>
>>>>>>>> "SET bytea_output=escape;"
>>>>>>>>
>>>>>>>> "SET client_encoding='UNICODE';")
>>>>>>>>
>>>>>>>
>>>>>>> It was probably before you joined, but I have said a number of times
>>>>>>> that pgAdmin 3 handled this differently and that maybe we should do it the
>>>>>>> same way here. See https://git.postgresql.org
>>>>>>> /gitweb/?p=pgadmin3.git;a=blob;f=pgadmin/db/pgConn.cpp, in the
>>>>>>> pgConn::Initialize() function.
>>>>>>>
>>>>>>> Either way, your patch isn't working for me.
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Note that this testing was on Python 2.7.10 on MacOS.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Kindly review.
>>>>>>>>>>
>>>>>>>>>> Thanks and Regards,
>>>>>>>>>> Aditya Toshniwal
>>>>>>>>>> Software Engineer | EnterpriseDB Software Solutions | Pune
>>>>>>>>>> "Don't Complain about Heat, Plant a tree"
>>>>>>>>>>
>>>>>>>>>> On Tue, Jun 5, 2018 at 10:15 AM, Aditya Toshniwal <
>>>>>>>>>> [email protected]> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Jun 5, 2018 at 1:08 AM, Joao De Almeida Pereira <
>>>>>>>>>>> [email protected]> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hello Aditya,
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> There is no change related to notifications in this patch.
>>>>>>>>>>>>> The below code is minor fix related to connection status of
>>>>>>>>>>>>> sql editor. Can you please share the code snippet if it is not the below.
>>>>>>>>>>>>>
>>>>>>>>>>>>> - # Check for the asynchronous notifies statements.
>>>>>>>>>>>>> - conn.check_notifies(True)
>>>>>>>>>>>>> - notifies = conn.get_notifies()
>>>>>>>>>>>>> + if status is not None:
>>>>>>>>>>>>> + # Check for the asynchronous notifies statements.
>>>>>>>>>>>>> + conn.check_notifies(True)
>>>>>>>>>>>>> + notifies = conn.get_notifies()
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>> This is a minor fix, but is it related to querying SQL_ASCII
>>>>>>>>>>>> database?
>>>>>>>>>>>>
>>>>>>>>>>> No its not. It is something I found when I was working on
>>>>>>>>>>> SQL_ASCII related changes.
>>>>>>>>>>> Well then, will send a separate patch for it.
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks
>>>>>>>>>>>> Victoria && Joao
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Dave Page
>>>>>>>>> Blog: http://pgsnake.blogspot.com
>>>>>>>>> Twitter: @pgsnake
>>>>>>>>>
>>>>>>>>> EnterpriseDB UK: http://www.enterprisedb.com
>>>>>>>>> The Enterprise PostgreSQL Company
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Dave Page
>>>>>>> Blog: http://pgsnake.blogspot.com
>>>>>>> Twitter: @pgsnake
>>>>>>>
>>>>>>> EnterpriseDB UK: http://www.enterprisedb.com
>>>>>>> The Enterprise PostgreSQL Company
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Dave Page
>>>>> Blog: http://pgsnake.blogspot.com
>>>>> Twitter: @pgsnake
>>>>>
>>>>> EnterpriseDB UK: http://www.enterprisedb.com
>>>>> The Enterprise PostgreSQL Company
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Dave Page
>>> Blog: http://pgsnake.blogspot.com
>>> Twitter: @pgsnake
>>>
>>> EnterpriseDB UK: http://www.enterprisedb.com
>>> The Enterprise PostgreSQL Company
>>>
>>
>>
>
>
> --
> Dave Page
> Blog: http://pgsnake.blogspot.com
> Twitter: @pgsnake
>
> EnterpriseDB UK: http://www.enterprisedb.com
> The Enterprise PostgreSQL Company
>
^ permalink raw reply [nested|flat] 38+ messages in thread
* Re: [pgAdmin4][RM#3289] Can't query SQL_ASCII database.
@ 2018-06-07 11:11 Dave Page <[email protected]>
parent: Aditya Toshniwal <[email protected]>
0 siblings, 1 reply; 38+ messages in thread
From: Dave Page @ 2018-06-07 11:11 UTC (permalink / raw)
To: Aditya Toshniwal <[email protected]>; +Cc: Joao De Almeida Pereira <[email protected]>; pgadmin-hackers
Hi
On Thu, Jun 7, 2018 at 12:05 PM, Aditya Toshniwal <
[email protected]> wrote:
> Hi Dave,
>
>
> On Thu, Jun 7, 2018 at 4:07 PM, Dave Page <[email protected]> wrote:
>
>> Hi
>>
>> On Wed, Jun 6, 2018 at 2:02 PM, Aditya Toshniwal <
>> [email protected]> wrote:
>>
>>> Hi Hackers,
>>>
>>> PFA updated patch as the previous one was not working as expected. I
>>> have tried to make it similar to that of pgAdmin3 and you do not need to
>>> change client_encoding as it is set now based on server encoding. It works
>>> fine with "view data" also.
>>>
>>
>> - In connection.py, at ~409, shouldn't we set the client_encoding to
>> SQL_ASCII? Otherwise it could be overridden with something unexpected if
>> the client has PGCLIENTENCODING set for example.
>>
> Yeah I agree, it would be better to add. Will add the change.
>
>>
>> - With or without that change, I get the following test failure on macOS
>> with Python 2.7.10:
>>
> It works fine on my machine with Python 2.7 and macOS. Could you please
> let me know the Postgres DB version also.
>
PostgreSQL 9.4.10 on x86_64-apple-darwin, compiled by
i686-apple-darwin11-llvm-gcc-4.2 (GCC) 4.2.1 (Based on Apple Inc. build
5658) (LLVM build 2336.11.00), 64-bit
> Will test on few more machines.
>
>>
>> ======================================================================
>> ERROR: runTest (pgadmin.tools.sqleditor.tests
>> .test_encoding_charset.TestEncodingCharset)
>> With Encoding SQL_ASCII
>> ----------------------------------------------------------------------
>> Traceback (most recent call last):
>> File "/Users/dpage/git/pgadmin4/web/pgadmin/tools/sqleditor/tests/test_encoding_charset.py",
>> line 86, in runTest
>> response = self.tester.get(url)
>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/werkzeug/test.py",
>> line 830, in get
>> return self.open(*args, **kw)
>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/testing.py",
>> line 127, in open
>> follow_redirects=follow_redirects)
>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/werkzeug/test.py",
>> line 803, in open
>> response = self.run_wsgi_app(environ, buffered=buffered)
>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/werkzeug/test.py",
>> line 716, in run_wsgi_app
>> rv = run_wsgi_app(self.application, environ, buffered=buffered)
>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/werkzeug/test.py",
>> line 923, in run_wsgi_app
>> app_rv = app(environ, start_response)
>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
>> line 1997, in __call__
>> return self.wsgi_app(environ, start_response)
>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
>> line 1985, in wsgi_app
>> response = self.handle_exception(e)
>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
>> line 1540, in handle_exception
>> reraise(exc_type, exc_value, tb)
>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
>> line 1982, in wsgi_app
>> response = self.full_dispatch_request()
>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
>> line 1614, in full_dispatch_request
>> rv = self.handle_user_exception(e)
>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
>> line 1517, in handle_user_exception
>> reraise(exc_type, exc_value, tb)
>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
>> line 1612, in full_dispatch_request
>> rv = self.dispatch_request()
>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
>> line 1598, in dispatch_request
>> return self.view_functions[rule.endpoint](**req.view_args)
>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask_login.py",
>> line 792, in decorated_view
>> return func(*args, **kwargs)
>> File "/Users/dpage/git/pgadmin4/web/pgadmin/tools/sqleditor/__init__.py",
>> line 576, in poll
>> 'oids': oids
>> File "/Users/dpage/git/pgadmin4/web/pgadmin/utils/ajax.py", line 61,
>> in make_json_response
>> separators=(',', ':')),
>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/simplejson/__init__.py",
>> line 399, in dumps
>> **kw).encode(obj)
>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/simplejson/encoder.py",
>> line 291, in encode
>> chunks = self.iterencode(o, _one_shot=True)
>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/simplejson/encoder.py",
>> line 373, in iterencode
>> return _iterencode(o, 0)
>> UnicodeDecodeError: 'utf8' codec can't decode byte 0xad in position 0:
>> invalid start byte
>>
>> ----------------------------------------------------------------------
>> Ran 317 tests in 30.692s
>>
>> FAILED (errors=1, skipped=21)
>>
>>
>>
>>> The only problem is, I cannot find equivalent codec for wxConvLibc in
>>> python. The closest one I could find is raw_unicode_escape. So, in a
>>> SQL_ASCII database, non ASCII characters may differ in pgAdmin4 and
>>> pgAdmin3, but it will display results.
>>>
>>
>> Yeah, I think that's fine. For the small number of people with SQL_ASCII
>> databases, seeing escaped characters is better than nothing.
>>
>>
>>>
>>>
>>> Dave,
>>> You need to add "E" before the string to be inserted, otherwise \x will
>>> be considered as a plain string.
>>> INSERT INTO sql_ascii (data) VALUES (E'[Invalid UTF-8] Blob:
>>> \xf4\xa5\xa3\xa5');
>>>
>>
>> Yeah, sorry - I copied the wrong version of the query :-(
>>
>>
>>>
>>>
>>> Kindly review.
>>>
>>> Thanks and Regards,
>>> Aditya Toshniwal
>>> Software Engineer | EnterpriseDB Software Solutions | Pune
>>> "Don't Complain about Heat, Plant a tree"
>>>
>>> On Tue, Jun 5, 2018 at 6:42 PM, Dave Page <[email protected]> wrote:
>>>
>>>> Hi
>>>>
>>>> On Tue, Jun 5, 2018 at 2:03 PM, Aditya Toshniwal <
>>>> [email protected]> wrote:
>>>>
>>>>> Hi
>>>>>
>>>>> On Tue, Jun 5, 2018 at 6:25 PM, Dave Page <[email protected]> wrote:
>>>>>
>>>>>>
>>>>>>
>>>>>> On Tue, Jun 5, 2018 at 1:49 PM, Aditya Toshniwal <
>>>>>> [email protected]> wrote:
>>>>>>
>>>>>>> Hi Dave,
>>>>>>>
>>>>>>> The problem of SQL ASCII is solved with the patch, and not related
>>>>>>> to setting the client encoding of the sql window.
>>>>>>>
>>>>>>
>>>>>> No it's not. It doesn't work for me as I said (and showed the example
>>>>>> of).
>>>>>>
>>>>>
>>>>> After setting the client_encoding to SQL_ASCII you got the output.
>>>>> Previously, it used to fail in the back end itself because python encoding
>>>>> failure. That is fixed.
>>>>> The error ERROR: invalid byte sequence for encoding "UTF8": 0x80 is
>>>>> thrown by postgres and not python or pgAdmin4. You will get the same error
>>>>> even if you
>>>>> connect from psql.
>>>>>
>>>>
>>>> Sure - but that is not a fix. You have no way of running the SET
>>>> command if you're using "view data" - and in the query tool, users just
>>>> expect it to work (as it did in pgAdmin 3).
>>>>
>>>>
>>>>>
>>>>>>
>>>>>>> I can see there is no SET call in pgAdmin3 for client_encoding. I
>>>>>>> can remove the SET client_encoding='UNICODE'; that will solve the
>>>>>>> problem. But, can you please let me know why that was added.
>>>>>>>
>>>>>>
>>>>>> There is, but it's inside an API call (PQsetClientEncoding):
>>>>>>
>>>>>> 300
>>>>>> <https://git.postgresql.org/gitweb/?p=pgadmin3.git;a=blob;f=pgadmin/db/pgConn.cpp#l300;
>>>>>> wxLogInfo(wxT("Setting client_encoding to '%s'")
>>>>>> , encoding.c_str());
>>>>>> 301
>>>>>> <https://git.postgresql.org/gitweb/?p=pgadmin3.git;a=blob;f=pgadmin/db/pgConn.cpp#l301;
>>>>>> if (PQsetClientEncoding(conn, encoding.ToAscii()))
>>>>>> 302
>>>>>> <https://git.postgresql.org/gitweb/?p=pgadmin3.git;a=blob;f=pgadmin/db/pgConn.cpp#l302;
>>>>>> {
>>>>>> 303
>>>>>> <https://git.postgresql.org/gitweb/?p=pgadmin3.git;a=blob;f=pgadmin/db/pgConn.cpp#l303;
>>>>>> wxLogError(wxT("%s"), GetLastError().c_str());
>>>>>> 304
>>>>>> <https://git.postgresql.org/gitweb/?p=pgadmin3.git;a=blob;f=pgadmin/db/pgConn.cpp#l304;
>>>>>> }
>>>>>> 305
>>>>>> <https://git.postgresql.org/gitweb/?p=pgadmin3.git;a=blob;f=pgadmin/db/pgConn.cpp#l305;
>>>>>>
>>>>>> Oops ! Missed that. Apologies.
>>>>>
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> Will remove the set call and will send you the updated patch if
>>>>>>> everything works fine.
>>>>>>>
>>>>>>
>>>>>> No, we need to ensure the client encoding is set correctly. It just
>>>>>> needs to be set to SQL_ASCII if it's a SQL_ASCII database (I believe).
>>>>>>
>>>>>>
>>>>> Need to rework on the initialise method. Will come with an updated.
>>>>> patch. Sorry for trouble.
>>>>>
>>>>>>
>>>>>>>
>>>>>>> On Tue, Jun 5, 2018 at 6:05 PM, Dave Page <[email protected]> wrote:
>>>>>>>
>>>>>>>> Hi
>>>>>>>>
>>>>>>>> On Tue, Jun 5, 2018 at 1:21 PM, Aditya Toshniwal <
>>>>>>>> [email protected]> wrote:
>>>>>>>>
>>>>>>>>> Hi Dave,
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Tue, Jun 5, 2018 at 4:56 PM, Dave Page <[email protected]>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hi
>>>>>>>>>>
>>>>>>>>>> On Tue, Jun 5, 2018 at 9:50 AM, Aditya Toshniwal <
>>>>>>>>>> [email protected]> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Hackers,
>>>>>>>>>>>
>>>>>>>>>>> PFA updated patch. The sqleditor change is sent separately and
>>>>>>>>>>> removed from current patch as suggested.
>>>>>>>>>>> The test cases were running fine when the module was specified
>>>>>>>>>>> using --pkg but were failing in complete run. Fixed that.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I did a quick test by creating a SQL_ASCII database containing a
>>>>>>>>>> simple table:
>>>>>>>>>>
>>>>>>>>>> CREATE TABLE sql_ascii (id serial primary key, data text);
>>>>>>>>>>
>>>>>>>>>> And then populated it with data:
>>>>>>>>>>
>>>>>>>>>> /Library/PostgreSQL/9.4/bin/psql -d sql_ascii -U postgres -c
>>>>>>>>>> "INSERT INTO sql_acsii (data) VALUES ('[Windows-1252] Euro: \x80 Double
>>>>>>>>>> dagger: \x87');"
>>>>>>>>>> /Library/PostgreSQL/9.4/bin/psql -d sql_ascii -U postgres -c
>>>>>>>>>> "INSERT INTO sql_ascii (data) VALUES ('[Latin-1] Yen: \xa5 Half:
>>>>>>>>>> \xbd');"
>>>>>>>>>> /Library/PostgreSQL/9.4/bin/psql -d sql_ascii -U postgres -c
>>>>>>>>>> "INSERT INTO sql_ascii (data) VALUES ('[Japanese] Ship: \xe8\x88\xb9');"
>>>>>>>>>> /Library/PostgreSQL/9.4/bin/psql -d sql_ascii -U postgres -c
>>>>>>>>>> "INSERT INTO sql_ascii (data) VALUES ('[Invalid UTF-8] Blob:
>>>>>>>>>> \xf4\xa5\xa3\xa5');"
>>>>>>>>>>
>>>>>>>>>> I then right-clicked the table in the treeview, and selected the
>>>>>>>>>> option to view all rows, and immediately saw an error:
>>>>>>>>>>
>>>>>>>>>> 2018-06-05 12:23:27,319: SQL pgadmin: Execute (async) for server
>>>>>>>>>> #1 - CONN:1187535 (Query-id: 8522474):
>>>>>>>>>> SELECT * FROM public.sql_ascii
>>>>>>>>>> ORDER BY id ASC
>>>>>>>>>> 2018-06-05 12:23:27,320: ERROR pgadmin: Failed to execute query
>>>>>>>>>> (execute_async) for the server #1 - CONN:1187535(Query-id: 8522474):
>>>>>>>>>> Error Message:ERROR: invalid byte sequence for encoding "UTF8":
>>>>>>>>>> 0x80
>>>>>>>>>> SQL state: 22021
>>>>>>>>>>
>>>>>>>>>> Running "SELECT * FROM sql_ascii" in the query tool resulted in
>>>>>>>>>> the same error, however, if I ran "SET client_encoding = 'SQL_ASCII';"
>>>>>>>>>> first, I do see results.
>>>>>>>>>>
>>>>>>>>>> I have confirmed that I've restarted the server after applying
>>>>>>>>>> the patch.
>>>>>>>>>>
>>>>>>>>>> What am I missing? Why don't we just set the client_encoding to
>>>>>>>>>> SQL_ASCII if it's a SQL_ASCII database?
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> It is by default same as the server encoding. But, the following
>>>>>>>>> existing code in web/pgadmin/utils/driver/psycopg2/connection.py makes
>>>>>>>>> the client_encoding as UNICODE for every connection. I am not sure it
>>>>>>>>> should be removed.
>>>>>>>>>
>>>>>>>>> status = _execute(cur, "SET DateStyle=ISO;"
>>>>>>>>>
>>>>>>>>> "SET client_min_messages=notice;"
>>>>>>>>>
>>>>>>>>> "SET bytea_output=escape;"
>>>>>>>>>
>>>>>>>>> "SET client_encoding='UNICODE';")
>>>>>>>>>
>>>>>>>>
>>>>>>>> It was probably before you joined, but I have said a number of
>>>>>>>> times that pgAdmin 3 handled this differently and that maybe we should do
>>>>>>>> it the same way here. See https://git.postgresql.org
>>>>>>>> /gitweb/?p=pgadmin3.git;a=blob;f=pgadmin/db/pgConn.cpp, in the
>>>>>>>> pgConn::Initialize() function.
>>>>>>>>
>>>>>>>> Either way, your patch isn't working for me.
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Note that this testing was on Python 2.7.10 on MacOS.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Kindly review.
>>>>>>>>>>>
>>>>>>>>>>> Thanks and Regards,
>>>>>>>>>>> Aditya Toshniwal
>>>>>>>>>>> Software Engineer | EnterpriseDB Software Solutions | Pune
>>>>>>>>>>> "Don't Complain about Heat, Plant a tree"
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Jun 5, 2018 at 10:15 AM, Aditya Toshniwal <
>>>>>>>>>>> [email protected]> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi
>>>>>>>>>>>>
>>>>>>>>>>>> On Tue, Jun 5, 2018 at 1:08 AM, Joao De Almeida Pereira <
>>>>>>>>>>>> [email protected]> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hello Aditya,
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> There is no change related to notifications in this patch.
>>>>>>>>>>>>>> The below code is minor fix related to connection status of
>>>>>>>>>>>>>> sql editor. Can you please share the code snippet if it is not the below.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> - # Check for the asynchronous notifies statements.
>>>>>>>>>>>>>> - conn.check_notifies(True)
>>>>>>>>>>>>>> - notifies = conn.get_notifies()
>>>>>>>>>>>>>> + if status is not None:
>>>>>>>>>>>>>> + # Check for the asynchronous notifies statements.
>>>>>>>>>>>>>> + conn.check_notifies(True)
>>>>>>>>>>>>>> + notifies = conn.get_notifies()
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>> This is a minor fix, but is it related to querying SQL_ASCII
>>>>>>>>>>>>> database?
>>>>>>>>>>>>>
>>>>>>>>>>>> No its not. It is something I found when I was working on
>>>>>>>>>>>> SQL_ASCII related changes.
>>>>>>>>>>>> Well then, will send a separate patch for it.
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>> Victoria && Joao
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Dave Page
>>>>>>>>>> Blog: http://pgsnake.blogspot.com
>>>>>>>>>> Twitter: @pgsnake
>>>>>>>>>>
>>>>>>>>>> EnterpriseDB UK: http://www.enterprisedb.com
>>>>>>>>>> The Enterprise PostgreSQL Company
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Dave Page
>>>>>>>> Blog: http://pgsnake.blogspot.com
>>>>>>>> Twitter: @pgsnake
>>>>>>>>
>>>>>>>> EnterpriseDB UK: http://www.enterprisedb.com
>>>>>>>> The Enterprise PostgreSQL Company
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Dave Page
>>>>>> Blog: http://pgsnake.blogspot.com
>>>>>> Twitter: @pgsnake
>>>>>>
>>>>>> EnterpriseDB UK: http://www.enterprisedb.com
>>>>>> The Enterprise PostgreSQL Company
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Dave Page
>>>> Blog: http://pgsnake.blogspot.com
>>>> Twitter: @pgsnake
>>>>
>>>> EnterpriseDB UK: http://www.enterprisedb.com
>>>> The Enterprise PostgreSQL Company
>>>>
>>>
>>>
>>
>>
>> --
>> Dave Page
>> Blog: http://pgsnake.blogspot.com
>> Twitter: @pgsnake
>>
>> EnterpriseDB UK: http://www.enterprisedb.com
>> The Enterprise PostgreSQL Company
>>
>
>
--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake
EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 38+ messages in thread
* Re: [pgAdmin4][RM#3289] Can't query SQL_ASCII database.
@ 2018-06-14 06:04 Aditya Toshniwal <[email protected]>
parent: Dave Page <[email protected]>
0 siblings, 1 reply; 38+ messages in thread
From: Aditya Toshniwal @ 2018-06-14 06:04 UTC (permalink / raw)
To: Dave Page <[email protected]>; +Cc: Joao De Almeida Pereira <[email protected]>; pgadmin-hackers
Hi Hackers,
PFA updated patch.
On Thu, Jun 7, 2018 at 4:41 PM, Dave Page <[email protected]> wrote:
> Hi
>
> On Thu, Jun 7, 2018 at 12:05 PM, Aditya Toshniwal <aditya.toshniwal@
> enterprisedb.com> wrote:
>
>> Hi Dave,
>>
>>
>> On Thu, Jun 7, 2018 at 4:07 PM, Dave Page <[email protected]> wrote:
>>
>>> Hi
>>>
>>> On Wed, Jun 6, 2018 at 2:02 PM, Aditya Toshniwal <
>>> [email protected]> wrote:
>>>
>>>> Hi Hackers,
>>>>
>>>> PFA updated patch as the previous one was not working as expected. I
>>>> have tried to make it similar to that of pgAdmin3 and you do not need to
>>>> change client_encoding as it is set now based on server encoding. It works
>>>> fine with "view data" also.
>>>>
>>>
>>> - In connection.py, at ~409, shouldn't we set the client_encoding to
>>> SQL_ASCII? Otherwise it could be overridden with something unexpected if
>>> the client has PGCLIENTENCODING set for example.
>>>
>> Yeah I agree, it would be better to add. Will add the change.
>>
>>>
>>> - With or without that change, I get the following test failure on macOS
>>> with Python 2.7.10:
>>>
>> It works fine on my machine with Python 2.7 and macOS. Could you please
>> let me know the Postgres DB version also.
>>
>
> PostgreSQL 9.4.10 on x86_64-apple-darwin, compiled by
> i686-apple-darwin11-llvm-gcc-4.2 (GCC) 4.2.1 (Based on Apple Inc. build
> 5658) (LLVM build 2336.11.00), 64-bit
>
>
>> Will test on few more machines.
>>
>>>
>>> ======================================================================
>>> ERROR: runTest (pgadmin.tools.sqleditor.tests
>>> .test_encoding_charset.TestEncodingCharset)
>>> With Encoding SQL_ASCII
>>> ----------------------------------------------------------------------
>>> Traceback (most recent call last):
>>> File "/Users/dpage/git/pgadmin4/web/pgadmin/tools/sqleditor/tests/test_encoding_charset.py",
>>> line 86, in runTest
>>> response = self.tester.get(url)
>>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/werkzeug/test.py",
>>> line 830, in get
>>> return self.open(*args, **kw)
>>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/testing.py",
>>> line 127, in open
>>> follow_redirects=follow_redirects)
>>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/werkzeug/test.py",
>>> line 803, in open
>>> response = self.run_wsgi_app(environ, buffered=buffered)
>>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/werkzeug/test.py",
>>> line 716, in run_wsgi_app
>>> rv = run_wsgi_app(self.application, environ, buffered=buffered)
>>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/werkzeug/test.py",
>>> line 923, in run_wsgi_app
>>> app_rv = app(environ, start_response)
>>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
>>> line 1997, in __call__
>>> return self.wsgi_app(environ, start_response)
>>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
>>> line 1985, in wsgi_app
>>> response = self.handle_exception(e)
>>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
>>> line 1540, in handle_exception
>>> reraise(exc_type, exc_value, tb)
>>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
>>> line 1982, in wsgi_app
>>> response = self.full_dispatch_request()
>>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
>>> line 1614, in full_dispatch_request
>>> rv = self.handle_user_exception(e)
>>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
>>> line 1517, in handle_user_exception
>>> reraise(exc_type, exc_value, tb)
>>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
>>> line 1612, in full_dispatch_request
>>> rv = self.dispatch_request()
>>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
>>> line 1598, in dispatch_request
>>> return self.view_functions[rule.endpoint](**req.view_args)
>>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask_login.py",
>>> line 792, in decorated_view
>>> return func(*args, **kwargs)
>>> File "/Users/dpage/git/pgadmin4/web/pgadmin/tools/sqleditor/__init__.py",
>>> line 576, in poll
>>> 'oids': oids
>>> File "/Users/dpage/git/pgadmin4/web/pgadmin/utils/ajax.py", line 61,
>>> in make_json_response
>>> separators=(',', ':')),
>>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/simplejson/__init__.py",
>>> line 399, in dumps
>>> **kw).encode(obj)
>>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/simplejson/encoder.py",
>>> line 291, in encode
>>> chunks = self.iterencode(o, _one_shot=True)
>>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/simplejson/encoder.py",
>>> line 373, in iterencode
>>> return _iterencode(o, 0)
>>> UnicodeDecodeError: 'utf8' codec can't decode byte 0xad in position 0:
>>> invalid start byte
>>>
>>> ----------------------------------------------------------------------
>>> Ran 317 tests in 30.692s
>>>
>>> FAILED (errors=1, skipped=21)
>>>
>>
This is fixed. There was a problem with json dumps. Now, json dumps will be
done based on connection encoding.
>
>>>
>>>
>>>> The only problem is, I cannot find equivalent codec for wxConvLibc in
>>>> python. The closest one I could find is raw_unicode_escape. So, in a
>>>> SQL_ASCII database, non ASCII characters may differ in pgAdmin4 and
>>>> pgAdmin3, but it will display results.
>>>>
>>>
>>> Yeah, I think that's fine. For the small number of people with SQL_ASCII
>>> databases, seeing escaped characters is better than nothing.
>>>
>>>
>>>>
>>>>
>>>> Dave,
>>>> You need to add "E" before the string to be inserted, otherwise \x will
>>>> be considered as a plain string.
>>>> INSERT INTO sql_ascii (data) VALUES (E'[Invalid UTF-8] Blob:
>>>> \xf4\xa5\xa3\xa5');
>>>>
>>>
>>> Yeah, sorry - I copied the wrong version of the query :-(
>>>
>>>
>>>>
>>>>
>>>> Kindly review.
>>>>
>>>> Thanks and Regards,
>>>> Aditya Toshniwal
>>>> Software Engineer | EnterpriseDB Software Solutions | Pune
>>>> "Don't Complain about Heat, Plant a tree"
>>>>
>>>> On Tue, Jun 5, 2018 at 6:42 PM, Dave Page <[email protected]> wrote:
>>>>
>>>>> Hi
>>>>>
>>>>> On Tue, Jun 5, 2018 at 2:03 PM, Aditya Toshniwal <
>>>>> [email protected]> wrote:
>>>>>
>>>>>> Hi
>>>>>>
>>>>>> On Tue, Jun 5, 2018 at 6:25 PM, Dave Page <[email protected]> wrote:
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Jun 5, 2018 at 1:49 PM, Aditya Toshniwal <
>>>>>>> [email protected]> wrote:
>>>>>>>
>>>>>>>> Hi Dave,
>>>>>>>>
>>>>>>>> The problem of SQL ASCII is solved with the patch, and not related
>>>>>>>> to setting the client encoding of the sql window.
>>>>>>>>
>>>>>>>
>>>>>>> No it's not. It doesn't work for me as I said (and showed the
>>>>>>> example of).
>>>>>>>
>>>>>>
>>>>>> After setting the client_encoding to SQL_ASCII you got the output.
>>>>>> Previously, it used to fail in the back end itself because python encoding
>>>>>> failure. That is fixed.
>>>>>> The error ERROR: invalid byte sequence for encoding "UTF8": 0x80 is
>>>>>> thrown by postgres and not python or pgAdmin4. You will get the same error
>>>>>> even if you
>>>>>> connect from psql.
>>>>>>
>>>>>
>>>>> Sure - but that is not a fix. You have no way of running the SET
>>>>> command if you're using "view data" - and in the query tool, users just
>>>>> expect it to work (as it did in pgAdmin 3).
>>>>>
>>>>>
>>>>>>
>>>>>>>
>>>>>>>> I can see there is no SET call in pgAdmin3 for client_encoding. I
>>>>>>>> can remove the SET client_encoding='UNICODE'; that will solve the
>>>>>>>> problem. But, can you please let me know why that was added.
>>>>>>>>
>>>>>>>
>>>>>>> There is, but it's inside an API call (PQsetClientEncoding):
>>>>>>>
>>>>>>> 300
>>>>>>> <https://git.postgresql.org/gitweb/?p=pgadmin3.git;a=blob;f=pgadmin/db/pgConn.cpp#l300;
>>>>>>> wxLogInfo(wxT("Setting client_encoding to '%s'")
>>>>>>> , encoding.c_str());
>>>>>>> 301
>>>>>>> <https://git.postgresql.org/gitweb/?p=pgadmin3.git;a=blob;f=pgadmin/db/pgConn.cpp#l301;
>>>>>>> if (PQsetClientEncoding(conn, encoding.ToAscii()))
>>>>>>> 302
>>>>>>> <https://git.postgresql.org/gitweb/?p=pgadmin3.git;a=blob;f=pgadmin/db/pgConn.cpp#l302;
>>>>>>> {
>>>>>>> 303
>>>>>>> <https://git.postgresql.org/gitweb/?p=pgadmin3.git;a=blob;f=pgadmin/db/pgConn.cpp#l303;
>>>>>>> wxLogError(wxT("%s"), GetLastError().c_str());
>>>>>>> 304
>>>>>>> <https://git.postgresql.org/gitweb/?p=pgadmin3.git;a=blob;f=pgadmin/db/pgConn.cpp#l304;
>>>>>>> }
>>>>>>> 305
>>>>>>> <https://git.postgresql.org/gitweb/?p=pgadmin3.git;a=blob;f=pgadmin/db/pgConn.cpp#l305;
>>>>>>>
>>>>>>> Oops ! Missed that. Apologies.
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> Will remove the set call and will send you the updated patch if
>>>>>>>> everything works fine.
>>>>>>>>
>>>>>>>
>>>>>>> No, we need to ensure the client encoding is set correctly. It just
>>>>>>> needs to be set to SQL_ASCII if it's a SQL_ASCII database (I believe).
>>>>>>>
>>>>>>>
>>>>>> Need to rework on the initialise method. Will come with an updated.
>>>>>> patch. Sorry for trouble.
>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, Jun 5, 2018 at 6:05 PM, Dave Page <[email protected]>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi
>>>>>>>>>
>>>>>>>>> On Tue, Jun 5, 2018 at 1:21 PM, Aditya Toshniwal <
>>>>>>>>> [email protected]> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Dave,
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Tue, Jun 5, 2018 at 4:56 PM, Dave Page <[email protected]>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Jun 5, 2018 at 9:50 AM, Aditya Toshniwal <
>>>>>>>>>>> [email protected]> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi Hackers,
>>>>>>>>>>>>
>>>>>>>>>>>> PFA updated patch. The sqleditor change is sent separately and
>>>>>>>>>>>> removed from current patch as suggested.
>>>>>>>>>>>> The test cases were running fine when the module was specified
>>>>>>>>>>>> using --pkg but were failing in complete run. Fixed that.
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I did a quick test by creating a SQL_ASCII database containing a
>>>>>>>>>>> simple table:
>>>>>>>>>>>
>>>>>>>>>>> CREATE TABLE sql_ascii (id serial primary key, data text);
>>>>>>>>>>>
>>>>>>>>>>> And then populated it with data:
>>>>>>>>>>>
>>>>>>>>>>> /Library/PostgreSQL/9.4/bin/psql -d sql_ascii -U postgres -c
>>>>>>>>>>> "INSERT INTO sql_acsii (data) VALUES ('[Windows-1252] Euro: \x80 Double
>>>>>>>>>>> dagger: \x87');"
>>>>>>>>>>> /Library/PostgreSQL/9.4/bin/psql -d sql_ascii -U postgres -c
>>>>>>>>>>> "INSERT INTO sql_ascii (data) VALUES ('[Latin-1] Yen: \xa5 Half:
>>>>>>>>>>> \xbd');"
>>>>>>>>>>> /Library/PostgreSQL/9.4/bin/psql -d sql_ascii -U postgres -c
>>>>>>>>>>> "INSERT INTO sql_ascii (data) VALUES ('[Japanese] Ship: \xe8\x88\xb9');"
>>>>>>>>>>> /Library/PostgreSQL/9.4/bin/psql -d sql_ascii -U postgres -c
>>>>>>>>>>> "INSERT INTO sql_ascii (data) VALUES ('[Invalid UTF-8] Blob:
>>>>>>>>>>> \xf4\xa5\xa3\xa5');"
>>>>>>>>>>>
>>>>>>>>>>> I then right-clicked the table in the treeview, and selected the
>>>>>>>>>>> option to view all rows, and immediately saw an error:
>>>>>>>>>>>
>>>>>>>>>>> 2018-06-05 12:23:27,319: SQL pgadmin: Execute (async) for
>>>>>>>>>>> server #1 - CONN:1187535 (Query-id: 8522474):
>>>>>>>>>>> SELECT * FROM public.sql_ascii
>>>>>>>>>>> ORDER BY id ASC
>>>>>>>>>>> 2018-06-05 12:23:27,320: ERROR pgadmin: Failed to execute query
>>>>>>>>>>> (execute_async) for the server #1 - CONN:1187535(Query-id: 8522474):
>>>>>>>>>>> Error Message:ERROR: invalid byte sequence for encoding "UTF8":
>>>>>>>>>>> 0x80
>>>>>>>>>>> SQL state: 22021
>>>>>>>>>>>
>>>>>>>>>>> Running "SELECT * FROM sql_ascii" in the query tool resulted in
>>>>>>>>>>> the same error, however, if I ran "SET client_encoding = 'SQL_ASCII';"
>>>>>>>>>>> first, I do see results.
>>>>>>>>>>>
>>>>>>>>>>> I have confirmed that I've restarted the server after applying
>>>>>>>>>>> the patch.
>>>>>>>>>>>
>>>>>>>>>>> What am I missing? Why don't we just set the client_encoding to
>>>>>>>>>>> SQL_ASCII if it's a SQL_ASCII database?
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> It is by default same as the server encoding. But, the following
>>>>>>>>>> existing code in web/pgadmin/utils/driver/psycopg2/connection.py
>>>>>>>>>> makes the client_encoding as UNICODE for every connection. I am
>>>>>>>>>> not sure it should be removed.
>>>>>>>>>>
>>>>>>>>>> status = _execute(cur, "SET DateStyle=ISO;"
>>>>>>>>>>
>>>>>>>>>> "SET client_min_messages=notice;"
>>>>>>>>>>
>>>>>>>>>> "SET bytea_output=escape;"
>>>>>>>>>>
>>>>>>>>>> "SET client_encoding='UNICODE';")
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> It was probably before you joined, but I have said a number of
>>>>>>>>> times that pgAdmin 3 handled this differently and that maybe we should do
>>>>>>>>> it the same way here. See https://git.postgresql.org
>>>>>>>>> /gitweb/?p=pgadmin3.git;a=blob;f=pgadmin/db/pgConn.cpp, in the
>>>>>>>>> pgConn::Initialize() function.
>>>>>>>>>
>>>>>>>>> Either way, your patch isn't working for me.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Note that this testing was on Python 2.7.10 on MacOS.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Kindly review.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks and Regards,
>>>>>>>>>>>> Aditya Toshniwal
>>>>>>>>>>>> Software Engineer | EnterpriseDB Software Solutions | Pune
>>>>>>>>>>>> "Don't Complain about Heat, Plant a tree"
>>>>>>>>>>>>
>>>>>>>>>>>> On Tue, Jun 5, 2018 at 10:15 AM, Aditya Toshniwal <
>>>>>>>>>>>> [email protected]> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Tue, Jun 5, 2018 at 1:08 AM, Joao De Almeida Pereira <
>>>>>>>>>>>>> [email protected]> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hello Aditya,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> There is no change related to notifications in this patch.
>>>>>>>>>>>>>>> The below code is minor fix related to connection status of
>>>>>>>>>>>>>>> sql editor. Can you please share the code snippet if it is not the below.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> - # Check for the asynchronous notifies statements.
>>>>>>>>>>>>>>> - conn.check_notifies(True)
>>>>>>>>>>>>>>> - notifies = conn.get_notifies()
>>>>>>>>>>>>>>> + if status is not None:
>>>>>>>>>>>>>>> + # Check for the asynchronous notifies
>>>>>>>>>>>>>>> statements.
>>>>>>>>>>>>>>> + conn.check_notifies(True)
>>>>>>>>>>>>>>> + notifies = conn.get_notifies()
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> This is a minor fix, but is it related to querying SQL_ASCII
>>>>>>>>>>>>>> database?
>>>>>>>>>>>>>>
>>>>>>>>>>>>> No its not. It is something I found when I was working on
>>>>>>>>>>>>> SQL_ASCII related changes.
>>>>>>>>>>>>> Well then, will send a separate patch for it.
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>> Victoria && Joao
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Dave Page
>>>>>>>>>>> Blog: http://pgsnake.blogspot.com
>>>>>>>>>>> Twitter: @pgsnake
>>>>>>>>>>>
>>>>>>>>>>> EnterpriseDB UK: http://www.enterprisedb.com
>>>>>>>>>>> The Enterprise PostgreSQL Company
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Dave Page
>>>>>>>>> Blog: http://pgsnake.blogspot.com
>>>>>>>>> Twitter: @pgsnake
>>>>>>>>>
>>>>>>>>> EnterpriseDB UK: http://www.enterprisedb.com
>>>>>>>>> The Enterprise PostgreSQL Company
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Dave Page
>>>>>>> Blog: http://pgsnake.blogspot.com
>>>>>>> Twitter: @pgsnake
>>>>>>>
>>>>>>> EnterpriseDB UK: http://www.enterprisedb.com
>>>>>>> The Enterprise PostgreSQL Company
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Dave Page
>>>>> Blog: http://pgsnake.blogspot.com
>>>>> Twitter: @pgsnake
>>>>>
>>>>> EnterpriseDB UK: http://www.enterprisedb.com
>>>>> The Enterprise PostgreSQL Company
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Dave Page
>>> Blog: http://pgsnake.blogspot.com
>>> Twitter: @pgsnake
>>>
>>> EnterpriseDB UK: http://www.enterprisedb.com
>>> The Enterprise PostgreSQL Company
>>>
>>
>>
>
>
> --
> Dave Page
> Blog: http://pgsnake.blogspot.com
> Twitter: @pgsnake
>
> EnterpriseDB UK: http://www.enterprisedb.com
> The Enterprise PostgreSQL Company
>
^ permalink raw reply [nested|flat] 38+ messages in thread
* Re: [pgAdmin4][RM#3289] Can't query SQL_ASCII database.
@ 2018-06-14 06:05 Aditya Toshniwal <[email protected]>
parent: Aditya Toshniwal <[email protected]>
0 siblings, 1 reply; 38+ messages in thread
From: Aditya Toshniwal @ 2018-06-14 06:05 UTC (permalink / raw)
To: Dave Page <[email protected]>; +Cc: Joao De Almeida Pereira <[email protected]>; pgadmin-hackers
I am sorry I missed the attachment. :(
PFA.
On Thu, Jun 14, 2018 at 11:34 AM, Aditya Toshniwal <
[email protected]> wrote:
> Hi Hackers,
>
> PFA updated patch.
>
> On Thu, Jun 7, 2018 at 4:41 PM, Dave Page <[email protected]> wrote:
>
>> Hi
>>
>> On Thu, Jun 7, 2018 at 12:05 PM, Aditya Toshniwal <
>> [email protected]> wrote:
>>
>>> Hi Dave,
>>>
>>>
>>> On Thu, Jun 7, 2018 at 4:07 PM, Dave Page <[email protected]> wrote:
>>>
>>>> Hi
>>>>
>>>> On Wed, Jun 6, 2018 at 2:02 PM, Aditya Toshniwal <
>>>> [email protected]> wrote:
>>>>
>>>>> Hi Hackers,
>>>>>
>>>>> PFA updated patch as the previous one was not working as expected. I
>>>>> have tried to make it similar to that of pgAdmin3 and you do not need to
>>>>> change client_encoding as it is set now based on server encoding. It works
>>>>> fine with "view data" also.
>>>>>
>>>>
>>>> - In connection.py, at ~409, shouldn't we set the client_encoding to
>>>> SQL_ASCII? Otherwise it could be overridden with something unexpected if
>>>> the client has PGCLIENTENCODING set for example.
>>>>
>>> Yeah I agree, it would be better to add. Will add the change.
>>>
>>>>
>>>> - With or without that change, I get the following test failure on
>>>> macOS with Python 2.7.10:
>>>>
>>> It works fine on my machine with Python 2.7 and macOS. Could you please
>>> let me know the Postgres DB version also.
>>>
>>
>> PostgreSQL 9.4.10 on x86_64-apple-darwin, compiled by
>> i686-apple-darwin11-llvm-gcc-4.2 (GCC) 4.2.1 (Based on Apple Inc. build
>> 5658) (LLVM build 2336.11.00), 64-bit
>>
>>
>>> Will test on few more machines.
>>>
>>>>
>>>> ======================================================================
>>>> ERROR: runTest (pgadmin.tools.sqleditor.tests
>>>> .test_encoding_charset.TestEncodingCharset)
>>>> With Encoding SQL_ASCII
>>>> ----------------------------------------------------------------------
>>>> Traceback (most recent call last):
>>>> File "/Users/dpage/git/pgadmin4/web/pgadmin/tools/sqleditor/tests/test_encoding_charset.py",
>>>> line 86, in runTest
>>>> response = self.tester.get(url)
>>>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/werkzeug/test.py",
>>>> line 830, in get
>>>> return self.open(*args, **kw)
>>>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/testing.py",
>>>> line 127, in open
>>>> follow_redirects=follow_redirects)
>>>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/werkzeug/test.py",
>>>> line 803, in open
>>>> response = self.run_wsgi_app(environ, buffered=buffered)
>>>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/werkzeug/test.py",
>>>> line 716, in run_wsgi_app
>>>> rv = run_wsgi_app(self.application, environ, buffered=buffered)
>>>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/werkzeug/test.py",
>>>> line 923, in run_wsgi_app
>>>> app_rv = app(environ, start_response)
>>>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
>>>> line 1997, in __call__
>>>> return self.wsgi_app(environ, start_response)
>>>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
>>>> line 1985, in wsgi_app
>>>> response = self.handle_exception(e)
>>>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
>>>> line 1540, in handle_exception
>>>> reraise(exc_type, exc_value, tb)
>>>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
>>>> line 1982, in wsgi_app
>>>> response = self.full_dispatch_request()
>>>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
>>>> line 1614, in full_dispatch_request
>>>> rv = self.handle_user_exception(e)
>>>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
>>>> line 1517, in handle_user_exception
>>>> reraise(exc_type, exc_value, tb)
>>>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
>>>> line 1612, in full_dispatch_request
>>>> rv = self.dispatch_request()
>>>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
>>>> line 1598, in dispatch_request
>>>> return self.view_functions[rule.endpoint](**req.view_args)
>>>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask_login.py",
>>>> line 792, in decorated_view
>>>> return func(*args, **kwargs)
>>>> File "/Users/dpage/git/pgadmin4/web/pgadmin/tools/sqleditor/__init__.py",
>>>> line 576, in poll
>>>> 'oids': oids
>>>> File "/Users/dpage/git/pgadmin4/web/pgadmin/utils/ajax.py", line 61,
>>>> in make_json_response
>>>> separators=(',', ':')),
>>>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/simplejson/__init__.py",
>>>> line 399, in dumps
>>>> **kw).encode(obj)
>>>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/simplejson/encoder.py",
>>>> line 291, in encode
>>>> chunks = self.iterencode(o, _one_shot=True)
>>>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/simplejson/encoder.py",
>>>> line 373, in iterencode
>>>> return _iterencode(o, 0)
>>>> UnicodeDecodeError: 'utf8' codec can't decode byte 0xad in position 0:
>>>> invalid start byte
>>>>
>>>> ----------------------------------------------------------------------
>>>> Ran 317 tests in 30.692s
>>>>
>>>> FAILED (errors=1, skipped=21)
>>>>
>>>
> This is fixed. There was a problem with json dumps. Now, json dumps will
> be done based on connection encoding.
>
>>
>>>>
>>>>
>>>>> The only problem is, I cannot find equivalent codec for wxConvLibc in
>>>>> python. The closest one I could find is raw_unicode_escape. So, in a
>>>>> SQL_ASCII database, non ASCII characters may differ in pgAdmin4 and
>>>>> pgAdmin3, but it will display results.
>>>>>
>>>>
>>>> Yeah, I think that's fine. For the small number of people with
>>>> SQL_ASCII databases, seeing escaped characters is better than nothing.
>>>>
>>>>
>>>>>
>>>>>
>>>>> Dave,
>>>>> You need to add "E" before the string to be inserted, otherwise \x
>>>>> will be considered as a plain string.
>>>>> INSERT INTO sql_ascii (data) VALUES (E'[Invalid UTF-8] Blob:
>>>>> \xf4\xa5\xa3\xa5');
>>>>>
>>>>
>>>> Yeah, sorry - I copied the wrong version of the query :-(
>>>>
>>>>
>>>>>
>>>>>
>>>>> Kindly review.
>>>>>
>>>>> Thanks and Regards,
>>>>> Aditya Toshniwal
>>>>> Software Engineer | EnterpriseDB Software Solutions | Pune
>>>>> "Don't Complain about Heat, Plant a tree"
>>>>>
>>>>> On Tue, Jun 5, 2018 at 6:42 PM, Dave Page <[email protected]> wrote:
>>>>>
>>>>>> Hi
>>>>>>
>>>>>> On Tue, Jun 5, 2018 at 2:03 PM, Aditya Toshniwal <
>>>>>> [email protected]> wrote:
>>>>>>
>>>>>>> Hi
>>>>>>>
>>>>>>> On Tue, Jun 5, 2018 at 6:25 PM, Dave Page <[email protected]> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, Jun 5, 2018 at 1:49 PM, Aditya Toshniwal <
>>>>>>>> [email protected]> wrote:
>>>>>>>>
>>>>>>>>> Hi Dave,
>>>>>>>>>
>>>>>>>>> The problem of SQL ASCII is solved with the patch, and not related
>>>>>>>>> to setting the client encoding of the sql window.
>>>>>>>>>
>>>>>>>>
>>>>>>>> No it's not. It doesn't work for me as I said (and showed the
>>>>>>>> example of).
>>>>>>>>
>>>>>>>
>>>>>>> After setting the client_encoding to SQL_ASCII you got the output.
>>>>>>> Previously, it used to fail in the back end itself because python encoding
>>>>>>> failure. That is fixed.
>>>>>>> The error ERROR: invalid byte sequence for encoding "UTF8": 0x80 is
>>>>>>> thrown by postgres and not python or pgAdmin4. You will get the same error
>>>>>>> even if you
>>>>>>> connect from psql.
>>>>>>>
>>>>>>
>>>>>> Sure - but that is not a fix. You have no way of running the SET
>>>>>> command if you're using "view data" - and in the query tool, users just
>>>>>> expect it to work (as it did in pgAdmin 3).
>>>>>>
>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>>> I can see there is no SET call in pgAdmin3 for client_encoding. I
>>>>>>>>> can remove the SET client_encoding='UNICODE'; that will solve the
>>>>>>>>> problem. But, can you please let me know why that was added.
>>>>>>>>>
>>>>>>>>
>>>>>>>> There is, but it's inside an API call (PQsetClientEncoding):
>>>>>>>>
>>>>>>>> 300
>>>>>>>> <https://git.postgresql.org/gitweb/?p=pgadmin3.git;a=blob;f=pgadmin/db/pgConn.cpp#l300;
>>>>>>>> wxLogInfo(wxT("Setting client_encoding to '%s'")
>>>>>>>> , encoding.c_str());
>>>>>>>> 301
>>>>>>>> <https://git.postgresql.org/gitweb/?p=pgadmin3.git;a=blob;f=pgadmin/db/pgConn.cpp#l301;
>>>>>>>> if (PQsetClientEncoding(conn, encoding.ToAscii()))
>>>>>>>> 302
>>>>>>>> <https://git.postgresql.org/gitweb/?p=pgadmin3.git;a=blob;f=pgadmin/db/pgConn.cpp#l302;
>>>>>>>> {
>>>>>>>> 303
>>>>>>>> <https://git.postgresql.org/gitweb/?p=pgadmin3.git;a=blob;f=pgadmin/db/pgConn.cpp#l303;
>>>>>>>> wxLogError(wxT("%s"), GetLastError().c_str());
>>>>>>>> 304
>>>>>>>> <https://git.postgresql.org/gitweb/?p=pgadmin3.git;a=blob;f=pgadmin/db/pgConn.cpp#l304;
>>>>>>>> }
>>>>>>>> 305
>>>>>>>> <https://git.postgresql.org/gitweb/?p=pgadmin3.git;a=blob;f=pgadmin/db/pgConn.cpp#l305;
>>>>>>>>
>>>>>>>> Oops ! Missed that. Apologies.
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Will remove the set call and will send you the updated patch if
>>>>>>>>> everything works fine.
>>>>>>>>>
>>>>>>>>
>>>>>>>> No, we need to ensure the client encoding is set correctly. It just
>>>>>>>> needs to be set to SQL_ASCII if it's a SQL_ASCII database (I believe).
>>>>>>>>
>>>>>>>>
>>>>>>> Need to rework on the initialise method. Will come with an updated.
>>>>>>> patch. Sorry for trouble.
>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Tue, Jun 5, 2018 at 6:05 PM, Dave Page <[email protected]>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hi
>>>>>>>>>>
>>>>>>>>>> On Tue, Jun 5, 2018 at 1:21 PM, Aditya Toshniwal <
>>>>>>>>>> [email protected]> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Dave,
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Jun 5, 2018 at 4:56 PM, Dave Page <[email protected]>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi
>>>>>>>>>>>>
>>>>>>>>>>>> On Tue, Jun 5, 2018 at 9:50 AM, Aditya Toshniwal <
>>>>>>>>>>>> [email protected]> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Hackers,
>>>>>>>>>>>>>
>>>>>>>>>>>>> PFA updated patch. The sqleditor change is sent separately and
>>>>>>>>>>>>> removed from current patch as suggested.
>>>>>>>>>>>>> The test cases were running fine when the module was specified
>>>>>>>>>>>>> using --pkg but were failing in complete run. Fixed that.
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> I did a quick test by creating a SQL_ASCII database containing
>>>>>>>>>>>> a simple table:
>>>>>>>>>>>>
>>>>>>>>>>>> CREATE TABLE sql_ascii (id serial primary key, data text);
>>>>>>>>>>>>
>>>>>>>>>>>> And then populated it with data:
>>>>>>>>>>>>
>>>>>>>>>>>> /Library/PostgreSQL/9.4/bin/psql -d sql_ascii -U postgres -c
>>>>>>>>>>>> "INSERT INTO sql_acsii (data) VALUES ('[Windows-1252] Euro: \x80 Double
>>>>>>>>>>>> dagger: \x87');"
>>>>>>>>>>>> /Library/PostgreSQL/9.4/bin/psql -d sql_ascii -U postgres -c
>>>>>>>>>>>> "INSERT INTO sql_ascii (data) VALUES ('[Latin-1] Yen: \xa5 Half:
>>>>>>>>>>>> \xbd');"
>>>>>>>>>>>> /Library/PostgreSQL/9.4/bin/psql -d sql_ascii -U postgres -c
>>>>>>>>>>>> "INSERT INTO sql_ascii (data) VALUES ('[Japanese] Ship: \xe8\x88\xb9');"
>>>>>>>>>>>> /Library/PostgreSQL/9.4/bin/psql -d sql_ascii -U postgres -c
>>>>>>>>>>>> "INSERT INTO sql_ascii (data) VALUES ('[Invalid UTF-8] Blob:
>>>>>>>>>>>> \xf4\xa5\xa3\xa5');"
>>>>>>>>>>>>
>>>>>>>>>>>> I then right-clicked the table in the treeview, and selected
>>>>>>>>>>>> the option to view all rows, and immediately saw an error:
>>>>>>>>>>>>
>>>>>>>>>>>> 2018-06-05 12:23:27,319: SQL pgadmin: Execute (async) for
>>>>>>>>>>>> server #1 - CONN:1187535 (Query-id: 8522474):
>>>>>>>>>>>> SELECT * FROM public.sql_ascii
>>>>>>>>>>>> ORDER BY id ASC
>>>>>>>>>>>> 2018-06-05 12:23:27,320: ERROR pgadmin: Failed to execute
>>>>>>>>>>>> query (execute_async) for the server #1 - CONN:1187535(Query-id: 8522474):
>>>>>>>>>>>> Error Message:ERROR: invalid byte sequence for encoding
>>>>>>>>>>>> "UTF8": 0x80
>>>>>>>>>>>> SQL state: 22021
>>>>>>>>>>>>
>>>>>>>>>>>> Running "SELECT * FROM sql_ascii" in the query tool resulted in
>>>>>>>>>>>> the same error, however, if I ran "SET client_encoding = 'SQL_ASCII';"
>>>>>>>>>>>> first, I do see results.
>>>>>>>>>>>>
>>>>>>>>>>>> I have confirmed that I've restarted the server after applying
>>>>>>>>>>>> the patch.
>>>>>>>>>>>>
>>>>>>>>>>>> What am I missing? Why don't we just set the client_encoding to
>>>>>>>>>>>> SQL_ASCII if it's a SQL_ASCII database?
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> It is by default same as the server encoding. But, the following
>>>>>>>>>>> existing code in web/pgadmin/utils/driver/psycopg2/connection.py
>>>>>>>>>>> makes the client_encoding as UNICODE for every connection. I am
>>>>>>>>>>> not sure it should be removed.
>>>>>>>>>>>
>>>>>>>>>>> status = _execute(cur, "SET DateStyle=ISO;"
>>>>>>>>>>>
>>>>>>>>>>> "SET client_min_messages=notice;"
>>>>>>>>>>>
>>>>>>>>>>> "SET bytea_output=escape;"
>>>>>>>>>>>
>>>>>>>>>>> "SET client_encoding='UNICODE';")
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> It was probably before you joined, but I have said a number of
>>>>>>>>>> times that pgAdmin 3 handled this differently and that maybe we should do
>>>>>>>>>> it the same way here. See https://git.postgresql.org
>>>>>>>>>> /gitweb/?p=pgadmin3.git;a=blob;f=pgadmin/db/pgConn.cpp, in the
>>>>>>>>>> pgConn::Initialize() function.
>>>>>>>>>>
>>>>>>>>>> Either way, your patch isn't working for me.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Note that this testing was on Python 2.7.10 on MacOS.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Kindly review.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks and Regards,
>>>>>>>>>>>>> Aditya Toshniwal
>>>>>>>>>>>>> Software Engineer | EnterpriseDB Software Solutions | Pune
>>>>>>>>>>>>> "Don't Complain about Heat, Plant a tree"
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Tue, Jun 5, 2018 at 10:15 AM, Aditya Toshniwal <
>>>>>>>>>>>>> [email protected]> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Tue, Jun 5, 2018 at 1:08 AM, Joao De Almeida Pereira <
>>>>>>>>>>>>>> [email protected]> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hello Aditya,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> There is no change related to notifications in this patch.
>>>>>>>>>>>>>>>> The below code is minor fix related to connection status of
>>>>>>>>>>>>>>>> sql editor. Can you please share the code snippet if it is not the below.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> - # Check for the asynchronous notifies statements.
>>>>>>>>>>>>>>>> - conn.check_notifies(True)
>>>>>>>>>>>>>>>> - notifies = conn.get_notifies()
>>>>>>>>>>>>>>>> + if status is not None:
>>>>>>>>>>>>>>>> + # Check for the asynchronous notifies
>>>>>>>>>>>>>>>> statements.
>>>>>>>>>>>>>>>> + conn.check_notifies(True)
>>>>>>>>>>>>>>>> + notifies = conn.get_notifies()
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> This is a minor fix, but is it related to querying SQL_ASCII
>>>>>>>>>>>>>>> database?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> No its not. It is something I found when I was working on
>>>>>>>>>>>>>> SQL_ASCII related changes.
>>>>>>>>>>>>>> Well then, will send a separate patch for it.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>> Victoria && Joao
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Dave Page
>>>>>>>>>>>> Blog: http://pgsnake.blogspot.com
>>>>>>>>>>>> Twitter: @pgsnake
>>>>>>>>>>>>
>>>>>>>>>>>> EnterpriseDB UK: http://www.enterprisedb.com
>>>>>>>>>>>> The Enterprise PostgreSQL Company
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Dave Page
>>>>>>>>>> Blog: http://pgsnake.blogspot.com
>>>>>>>>>> Twitter: @pgsnake
>>>>>>>>>>
>>>>>>>>>> EnterpriseDB UK: http://www.enterprisedb.com
>>>>>>>>>> The Enterprise PostgreSQL Company
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Dave Page
>>>>>>>> Blog: http://pgsnake.blogspot.com
>>>>>>>> Twitter: @pgsnake
>>>>>>>>
>>>>>>>> EnterpriseDB UK: http://www.enterprisedb.com
>>>>>>>> The Enterprise PostgreSQL Company
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Dave Page
>>>>>> Blog: http://pgsnake.blogspot.com
>>>>>> Twitter: @pgsnake
>>>>>>
>>>>>> EnterpriseDB UK: http://www.enterprisedb.com
>>>>>> The Enterprise PostgreSQL Company
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Dave Page
>>>> Blog: http://pgsnake.blogspot.com
>>>> Twitter: @pgsnake
>>>>
>>>> EnterpriseDB UK: http://www.enterprisedb.com
>>>> The Enterprise PostgreSQL Company
>>>>
>>>
>>>
>>
>>
>> --
>> Dave Page
>> Blog: http://pgsnake.blogspot.com
>> Twitter: @pgsnake
>>
>> EnterpriseDB UK: http://www.enterprisedb.com
>> The Enterprise PostgreSQL Company
>>
>
>
Attachments:
[application/octet-stream] RM3289.patch (11.5K, 3-RM3289.patch)
download | inline diff:
diff --git a/web/pgadmin/tools/sqleditor/__init__.py b/web/pgadmin/tools/sqleditor/__init__.py
index a9460dd7..d8f2ce67 100644
--- a/web/pgadmin/tools/sqleditor/__init__.py
+++ b/web/pgadmin/tools/sqleditor/__init__.py
@@ -574,7 +574,8 @@ def poll(trans_id):
'client_primary_key': client_primary_key,
'has_oids': has_oids,
'oids': oids
- }
+ },
+ encoding=conn.python_encoding
)
@@ -646,7 +647,8 @@ def fetch(trans_id, fetch_all=None):
'has_more_rows': has_more_rows,
'rows_fetched_from': rows_fetched_from,
'rows_fetched_to': rows_fetched_to
- }
+ },
+ encoding=conn.python_encoding
)
diff --git a/web/pgadmin/tools/sqleditor/tests/test_encoding_charset.py b/web/pgadmin/tools/sqleditor/tests/test_encoding_charset.py
new file mode 100644
index 00000000..0826decb
--- /dev/null
+++ b/web/pgadmin/tools/sqleditor/tests/test_encoding_charset.py
@@ -0,0 +1,103 @@
+##########################################################################
+#
+# pgAdmin 4 - PostgreSQL Tools
+#
+# Copyright (C) 2013 - 2018, The pgAdmin Development Team
+# This software is released under the PostgreSQL Licence
+#
+##########################################################################
+
+from pgadmin.utils.route import BaseTestGenerator
+from pgadmin.browser.server_groups.servers.databases.tests import utils as \
+ database_utils
+from regression import parent_node_dict
+from regression.python_test_utils import test_utils
+import json
+
+
+class TestEncodingCharset(BaseTestGenerator):
+ """
+ This class validates character support in pgAdmin4 for
+ different PostgresDB encodings
+ """
+ scenarios = [
+ (
+ 'With Encoding UTF8',
+ dict(
+ db_encoding='UTF8',
+ lc_collate='C',
+ test_str='A'
+ )),
+ (
+ 'With Encoding WIN1252',
+ dict(
+ db_encoding='WIN1252',
+ lc_collate='C',
+ test_str='A'
+ )),
+ (
+ 'With Encoding EUC_CN',
+ dict(
+ db_encoding='EUC_CN',
+ lc_collate='C',
+ test_str='A'
+ )),
+ (
+ 'With Encoding SQL_ASCII',
+ dict(
+ db_encoding='SQL_ASCII',
+ lc_collate='C',
+ test_str='\\255'
+ )),
+ ]
+
+ def setUp(self):
+ self.encode_db_name = 'encoding_' + self.db_encoding
+ self.encode_sid = self.server_information['server_id']
+ self.encode_did = test_utils.create_database(
+ self.server, self.encode_db_name,
+ (self.db_encoding, self.lc_collate))
+
+ def runTest(self):
+
+ db_con = database_utils.connect_database(self,
+ test_utils.SERVER_GROUP,
+ self.encode_sid,
+ self.encode_did)
+ if not db_con["info"] == "Database connected.":
+ raise Exception("Could not connect to the database.")
+
+ # Initialize query tool
+ url = '/datagrid/initialize/query_tool/{0}/{1}/{2}'.format(
+ test_utils.SERVER_GROUP, self.encode_sid, self.encode_did)
+ response = self.tester.post(url)
+ self.assertEquals(response.status_code, 200)
+
+ response_data = json.loads(response.data.decode('utf-8'))
+ self.trans_id = response_data['data']['gridTransId']
+
+ # Check character
+ url = "/sqleditor/query_tool/start/{0}".format(self.trans_id)
+ sql = "select E'{0}';".format(self.test_str)
+ response = self.tester.post(url, data=json.dumps({"sql": sql}),
+ content_type='html/json')
+ self.assertEquals(response.status_code, 200)
+ url = '/sqleditor/poll/{0}'.format(self.trans_id)
+ response = self.tester.get(url)
+ self.assertEquals(response.status_code, 200)
+ response_data = json.loads(response.data.decode('utf-8'))
+ self.assertEquals(response_data['data']['rows_fetched_to'], 1)
+
+ database_utils.disconnect_database(self, self.encode_sid,
+ self.encode_did)
+
+ def tearDown(self):
+ main_conn = test_utils.get_db_connection(
+ self.server['db'],
+ self.server['username'],
+ self.server['db_password'],
+ self.server['host'],
+ self.server['port'],
+ self.server['sslmode']
+ )
+ test_utils.drop_database(main_conn, self.encode_db_name)
diff --git a/web/pgadmin/utils/ajax.py b/web/pgadmin/utils/ajax.py
index 9b44aa3e..c19f778a 100644
--- a/web/pgadmin/utils/ajax.py
+++ b/web/pgadmin/utils/ajax.py
@@ -45,7 +45,8 @@ def get_no_cache_header():
def make_json_response(
- success=1, errormsg='', info='', result=None, data=None, status=200
+ success=1, errormsg='', info='', result=None, data=None, status=200,
+ encoding='utf-8'
):
"""Create a HTML response document describing the results of a request and
containing the data."""
@@ -58,7 +59,7 @@ def make_json_response(
return Response(
response=json.dumps(doc, cls=DataTypeJSONEncoder,
- separators=(',', ':')),
+ separators=(',', ':'), encoding=encoding),
status=status,
mimetype="application/json",
headers=get_no_cache_header()
diff --git a/web/pgadmin/utils/driver/psycopg2/connection.py b/web/pgadmin/utils/driver/psycopg2/connection.py
index cfd161a0..33762558 100644
--- a/web/pgadmin/utils/driver/psycopg2/connection.py
+++ b/web/pgadmin/utils/driver/psycopg2/connection.py
@@ -33,7 +33,7 @@ from pgadmin.utils import get_complete_file_path
from ..abstract import BaseDriver, BaseConnection
from .cursor import DictCursor
from .typecast import register_global_typecasters, \
- register_string_typecasters, register_binary_typecasters, \
+ register_binary_typecasters, \
register_array_to_string_typecasters, ALL_JSON_TYPES
@@ -387,8 +387,6 @@ class Connection(BaseConnection):
else:
self.conn.autocommit = True
- register_string_typecasters(self.conn)
-
if self.array_to_string:
register_array_to_string_typecasters(self.conn)
@@ -397,11 +395,22 @@ class Connection(BaseConnection):
if self.use_binary_placeholder:
register_binary_typecasters(self.conn)
- status = _execute(cur, "SET DateStyle=ISO;"
- "SET client_min_messages=notice;"
- "SET bytea_output=escape;"
- "SET client_encoding='UNICODE';")
-
+ if self.conn.encoding not in ('SQL_ASCII', 'SQLASCII',
+ 'MULE_INTERNAL', 'MULEINTERNAL'):
+ status = _execute(cur, "SET DateStyle=ISO;"
+ "SET client_min_messages=notice;"
+ "SET bytea_output=escape;"
+ "SET client_encoding='UNICODE';")
+ self.python_encoding = 'utf-8'
+ encodings[self.conn.encoding] = self.python_encoding
+ else:
+ status = _execute(cur, "SET DateStyle=ISO;"
+ "SET client_min_messages=notice;"
+ "SET bytea_output=escape;"
+ "SET client_encoding='{0}';"
+ .format(self.conn.encoding))
+ self.python_encoding = 'raw_unicode_escape'
+ encodings[self.conn.encoding] = self.python_encoding
if status is not None:
self.conn.close()
self.conn = None
diff --git a/web/pgadmin/utils/driver/psycopg2/typecast.py b/web/pgadmin/utils/driver/psycopg2/typecast.py
index f1366049..a8f6c385 100644
--- a/web/pgadmin/utils/driver/psycopg2/typecast.py
+++ b/web/pgadmin/utils/driver/psycopg2/typecast.py
@@ -163,49 +163,6 @@ def register_global_typecasters():
psycopg2.extensions.register_type(pg_array_types_to_array_of_string_type)
-def register_string_typecasters(connection):
- if connection.encoding != 'UTF8':
- # In python3 when database encoding is other than utf-8 and client
- # encoding is set to UNICODE then we need to map data from database
- # encoding to utf-8.
- # This is required because when client encoding is set to UNICODE then
- # psycopg assumes database encoding utf-8 and not the actual encoding.
- # Not sure whether it's bug or feature in psycopg for python3.
- if sys.version_info >= (3,):
- def return_as_unicode(value, cursor):
- if value is None:
- return None
- # Treat value as byte sequence of database encoding and then
- # decode it as utf-8 to get correct unicode value.
- return bytes(
- value, encodings[cursor.connection.encoding]
- ).decode('utf-8')
-
- unicode_type = psycopg2.extensions.new_type(
- # "char", name, text, character, character varying
- (19, 18, 25, 1042, 1043, 0),
- 'UNICODE', return_as_unicode)
- else:
- def return_as_unicode(value, cursor):
- if value is None:
- return None
- # Decode it as utf-8 to get correct unicode value.
- return value.decode('utf-8')
-
- unicode_type = psycopg2.extensions.new_type(
- # "char", name, text, character, character varying
- (19, 18, 25, 1042, 1043, 0),
- 'UNICODE', return_as_unicode)
-
- unicode_array_type = psycopg2.extensions.new_array_type(
- # "char"[], name[], text[], character[], character varying[]
- (1002, 1003, 1009, 1014, 1015, 0
- ), 'UNICODEARRAY', unicode_type)
-
- psycopg2.extensions.register_type(unicode_type)
- psycopg2.extensions.register_type(unicode_array_type)
-
-
def register_binary_typecasters(connection):
psycopg2.extensions.register_type(
psycopg2.extensions.new_type(
diff --git a/web/regression/python_test_utils/test_utils.py b/web/regression/python_test_utils/test_utils.py
index 3e517b61..464a09e1 100644
--- a/web/regression/python_test_utils/test_utils.py
+++ b/web/regression/python_test_utils/test_utils.py
@@ -116,7 +116,7 @@ def clear_node_info_dict():
del node_info_dict[node][:]
-def create_database(server, db_name):
+def create_database(server, db_name, encoding=None):
"""This function used to create database and returns the database id"""
try:
connection = get_db_connection(
@@ -130,8 +130,14 @@ def create_database(server, db_name):
old_isolation_level = connection.isolation_level
connection.set_isolation_level(0)
pg_cursor = connection.cursor()
- pg_cursor.execute(
- '''CREATE DATABASE "%s" TEMPLATE template0''' % db_name)
+ if encoding is None:
+ pg_cursor.execute(
+ '''CREATE DATABASE "%s" TEMPLATE template0''' % db_name)
+ else:
+ pg_cursor.execute(
+ '''CREATE DATABASE "%s" TEMPLATE template0
+ ENCODING='%s' LC_COLLATE='%s' LC_CTYPE='%s' ''' %
+ (db_name, encoding[0], encoding[1], encoding[1]))
connection.set_isolation_level(old_isolation_level)
connection.commit()
^ permalink raw reply [nested|flat] 38+ messages in thread
* Re: [pgAdmin4][RM#3289] Can't query SQL_ASCII database.
@ 2018-06-15 08:38 Dave Page <[email protected]>
parent: Aditya Toshniwal <[email protected]>
0 siblings, 1 reply; 38+ messages in thread
From: Dave Page @ 2018-06-15 08:38 UTC (permalink / raw)
To: Aditya Toshniwal <[email protected]>; +Cc: Joao De Almeida Pereira <[email protected]>; pgadmin-hackers
Hi
On Thu, Jun 14, 2018 at 7:05 AM, Aditya Toshniwal <
[email protected]> wrote:
> I am sorry I missed the attachment. :(
> PFA.
>
It looks like the encoding names are getting munged somewhere. I see you've
accounted for that to some degree in connection.py (you have both
SQL_ASCII/MULE_INTERNAL and SQLASCII/MULEINTERNAL), however it doesn't seem
to be enough as I'm getting the following error when trying to download CSV
from the query tool. Can we ensure that conn.encoding contains an un-munged
value at all times, or is that coming from psycopg2?
2018-06-15 09:32:28,799: INFO werkzeug: 127.0.0.1 - - [15/Jun/2018
09:32:28] "GET
/sqleditor/query_tool/download/2732923?query=SELECT%20*%20FROM%20public.sql_ascii%0AORDER%20BY%20id%20ASC%20&filename=sql_ascii.csv
HTTP/1.1" 500 -
2018-06-15 09:32:28,801: ERROR werkzeug: Error on request:
Traceback (most recent call last):
File
"/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/werkzeug/serving.py",
line 270, in run_wsgi
execute(self.server.app)
File
"/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/werkzeug/serving.py",
line 260, in execute
for data in application_iter:
File
"/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/werkzeug/wsgi.py",
line 870, in __next__
return self._next()
File
"/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/werkzeug/wrappers.py",
line 82, in _iter_encoded
for item in iterable:
File
"/Users/dpage/git/pgadmin4/web/pgadmin/utils/driver/psycopg2/connection.py",
line 752, in gen
column_name = column_name.decode(conn_encoding)
LookupError: unknown encoding: SQLASCII
--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake
EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 38+ messages in thread
* Re: [pgAdmin4][RM#3289] Can't query SQL_ASCII database.
@ 2018-06-15 09:03 Aditya Toshniwal <[email protected]>
parent: Dave Page <[email protected]>
0 siblings, 1 reply; 38+ messages in thread
From: Aditya Toshniwal @ 2018-06-15 09:03 UTC (permalink / raw)
To: Dave Page <[email protected]>; +Cc: Joao De Almeida Pereira <[email protected]>; pgadmin-hackers
Hi Dave,
On Fri, Jun 15, 2018 at 2:08 PM, Dave Page <[email protected]> wrote:
> Hi
>
> On Thu, Jun 14, 2018 at 7:05 AM, Aditya Toshniwal <aditya.toshniwal@
> enterprisedb.com> wrote:
>
>> I am sorry I missed the attachment. :(
>> PFA.
>>
>
> It looks like the encoding names are getting munged somewhere. I see
> you've accounted for that to some degree in connection.py (you have both
> SQL_ASCII/MULE_INTERNAL and SQLASCII/MULEINTERNAL), however it doesn't seem
> to be enough as I'm getting the following error when trying to download CSV
> from the query tool. Can we ensure that conn.encoding contains an un-munged
> value at all times, or is that coming from psycopg2?
>
That is done by pyscopg2 and conn.encoding is a psycopg2 connection
property.
>
> 2018-06-15 09:32:28,799: INFO werkzeug: 127.0.0.1 - - [15/Jun/2018
> 09:32:28] "GET /sqleditor/query_tool/download/2732923?query=SELECT%
> 20*%20FROM%20public.sql_ascii%0AORDER%20BY%20id%20ASC%20&filename=sql_ascii.csv
> HTTP/1.1" 500 -
> 2018-06-15 09:32:28,801: ERROR werkzeug: Error on request:
> Traceback (most recent call last):
> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/werkzeug/serving.py",
> line 270, in run_wsgi
> execute(self.server.app)
> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/werkzeug/serving.py",
> line 260, in execute
> for data in application_iter:
> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/werkzeug/wsgi.py",
> line 870, in __next__
> return self._next()
> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-
> packages/werkzeug/wrappers.py", line 82, in _iter_encoded
> for item in iterable:
> File "/Users/dpage/git/pgadmin4/web/pgadmin/utils/driver/psycopg2/connection.py",
> line 752, in gen
> column_name = column_name.decode(conn_encoding)
> LookupError: unknown encoding: SQLASCII
>
This is because there is code bug here. Below is code used to decode a
column name. Connection encoding and python encoding are two different
things. Python does not know what SQLASCII is. This will work with UTF-8
because python has decoder with same name. I tried to download CSV with the
original code without changes and it fails there too. I will fix this and
will send the updated patch. I should have checked this.
conn_encoding = cur.connection.encoding
column_name = column_name.decode(conn_encoding)
>
> --
> Dave Page
> Blog: http://pgsnake.blogspot.com
> Twitter: @pgsnake
>
> EnterpriseDB UK: http://www.enterprisedb.com
> The Enterprise PostgreSQL Company
>
--
Thanks and Regards,
Aditya Toshniwal
Software Engineer | EnterpriseDB Software Solutions | Pune
"Don't Complain about Heat, Plant a tree"
^ permalink raw reply [nested|flat] 38+ messages in thread
* Re: [pgAdmin4][RM#3289] Can't query SQL_ASCII database.
@ 2018-06-18 13:14 Aditya Toshniwal <[email protected]>
parent: Aditya Toshniwal <[email protected]>
0 siblings, 1 reply; 38+ messages in thread
From: Aditya Toshniwal @ 2018-06-18 13:14 UTC (permalink / raw)
To: Dave Page <[email protected]>; +Cc: Joao De Almeida Pereira <[email protected]>; pgadmin-hackers
Hi Hackers,
Attached is the updated patch which includes the fix for Download CSV fail
in SQL_ASCII database, which is RM3250
<https://redmine.postgresql.org/issues/3250;
This should fix RM3289 and RM3250. As they interrelated, sending the
combined patch.
Kindly review.
On Fri, Jun 15, 2018 at 2:33 PM, Aditya Toshniwal <
[email protected]> wrote:
> Hi Dave,
>
> On Fri, Jun 15, 2018 at 2:08 PM, Dave Page <[email protected]> wrote:
>
>> Hi
>>
>> On Thu, Jun 14, 2018 at 7:05 AM, Aditya Toshniwal <
>> [email protected]> wrote:
>>
>>> I am sorry I missed the attachment. :(
>>> PFA.
>>>
>>
>> It looks like the encoding names are getting munged somewhere. I see
>> you've accounted for that to some degree in connection.py (you have both
>> SQL_ASCII/MULE_INTERNAL and SQLASCII/MULEINTERNAL), however it doesn't seem
>> to be enough as I'm getting the following error when trying to download CSV
>> from the query tool. Can we ensure that conn.encoding contains an un-munged
>> value at all times, or is that coming from psycopg2?
>>
> That is done by pyscopg2 and conn.encoding is a psycopg2 connection
> property.
>
>
>>
>> 2018-06-15 09:32:28,799: INFO werkzeug: 127.0.0.1 - - [15/Jun/2018
>> 09:32:28] "GET /sqleditor/query_tool/download/2732923?query=SELECT%20*%
>> 20FROM%20public.sql_ascii%0AORDER%20BY%20id%20ASC%20&filename=sql_ascii.csv
>> HTTP/1.1" 500 -
>> 2018-06-15 09:32:28,801: ERROR werkzeug: Error on request:
>> Traceback (most recent call last):
>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/werkzeug/serving.py",
>> line 270, in run_wsgi
>> execute(self.server.app)
>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/werkzeug/serving.py",
>> line 260, in execute
>> for data in application_iter:
>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/werkzeug/wsgi.py",
>> line 870, in __next__
>> return self._next()
>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/werkzeug/wrappers.py",
>> line 82, in _iter_encoded
>> for item in iterable:
>> File "/Users/dpage/git/pgadmin4/web/pgadmin/utils/driver/psycopg2/connection.py",
>> line 752, in gen
>> column_name = column_name.decode(conn_encoding)
>> LookupError: unknown encoding: SQLASCII
>>
>
> This is because there is code bug here. Below is code used to decode a
> column name. Connection encoding and python encoding are two different
> things. Python does not know what SQLASCII is. This will work with UTF-8
> because python has decoder with same name. I tried to download CSV with the
> original code without changes and it fails there too. I will fix this and
> will send the updated patch. I should have checked this.
> conn_encoding = cur.connection.encoding
> column_name = column_name.decode(conn_encoding)
>
>
>>
>> --
>> Dave Page
>> Blog: http://pgsnake.blogspot.com
>> Twitter: @pgsnake
>>
>> EnterpriseDB UK: http://www.enterprisedb.com
>> The Enterprise PostgreSQL Company
>>
>
>
>
> --
> Thanks and Regards,
> Aditya Toshniwal
> Software Engineer | EnterpriseDB Software Solutions | Pune
> "Don't Complain about Heat, Plant a tree"
>
--
Thanks and Regards,
Aditya Toshniwal
Software Engineer | EnterpriseDB Software Solutions | Pune
"Don't Complain about Heat, Plant a tree"
Attachments:
[application/octet-stream] RM3289_3250.patch (12.3K, 3-RM3289_3250.patch)
download | inline diff:
diff --git a/web/pgadmin/tools/sqleditor/__init__.py b/web/pgadmin/tools/sqleditor/__init__.py
index a9460dd7..d8f2ce67 100644
--- a/web/pgadmin/tools/sqleditor/__init__.py
+++ b/web/pgadmin/tools/sqleditor/__init__.py
@@ -574,7 +574,8 @@ def poll(trans_id):
'client_primary_key': client_primary_key,
'has_oids': has_oids,
'oids': oids
- }
+ },
+ encoding=conn.python_encoding
)
@@ -646,7 +647,8 @@ def fetch(trans_id, fetch_all=None):
'has_more_rows': has_more_rows,
'rows_fetched_from': rows_fetched_from,
'rows_fetched_to': rows_fetched_to
- }
+ },
+ encoding=conn.python_encoding
)
diff --git a/web/pgadmin/tools/sqleditor/tests/test_encoding_charset.py b/web/pgadmin/tools/sqleditor/tests/test_encoding_charset.py
new file mode 100644
index 00000000..8a8d45e3
--- /dev/null
+++ b/web/pgadmin/tools/sqleditor/tests/test_encoding_charset.py
@@ -0,0 +1,113 @@
+##########################################################################
+#
+# pgAdmin 4 - PostgreSQL Tools
+#
+# Copyright (C) 2013 - 2018, The pgAdmin Development Team
+# This software is released under the PostgreSQL Licence
+#
+##########################################################################
+
+from pgadmin.utils.route import BaseTestGenerator
+from pgadmin.browser.server_groups.servers.databases.tests import utils as \
+ database_utils
+from regression import parent_node_dict
+from regression.python_test_utils import test_utils
+import json
+from pgadmin.utils import server_utils
+
+
+class TestEncodingCharset(BaseTestGenerator):
+ """
+ This class validates character support in pgAdmin4 for
+ different PostgresDB encodings
+ """
+ skip_on_database = ['gpdb']
+ scenarios = [
+ (
+ 'With Encoding UTF8',
+ dict(
+ db_encoding='UTF8',
+ lc_collate='C',
+ test_str='A'
+ )),
+ (
+ 'With Encoding WIN1252',
+ dict(
+ db_encoding='WIN1252',
+ lc_collate='C',
+ test_str='A'
+ )),
+ (
+ 'With Encoding EUC_CN',
+ dict(
+ db_encoding='EUC_CN',
+ lc_collate='C',
+ test_str='A'
+ )),
+ (
+ 'With Encoding SQL_ASCII',
+ dict(
+ db_encoding='SQL_ASCII',
+ lc_collate='C',
+ test_str='\\255'
+ )),
+ ]
+
+ def setUp(self):
+ self.encode_db_name = 'encoding_' + self.db_encoding
+ self.encode_sid = self.server_information['server_id']
+
+ server_con = server_utils.connect_server(self, self.encode_sid)
+ if hasattr(self, 'skip_on_database'):
+ if 'data' in server_con and 'type' in server_con['data']:
+ if server_con['data']['type'] in self.skip_on_database:
+ self.skipTest('cannot run in: %s' %
+ server_con['data']['type'])
+
+ self.encode_did = test_utils.create_database(
+ self.server, self.encode_db_name,
+ (self.db_encoding, self.lc_collate))
+
+ def runTest(self):
+
+ db_con = database_utils.connect_database(self,
+ test_utils.SERVER_GROUP,
+ self.encode_sid,
+ self.encode_did)
+ if not db_con["info"] == "Database connected.":
+ raise Exception("Could not connect to the database.")
+
+ # Initialize query tool
+ url = '/datagrid/initialize/query_tool/{0}/{1}/{2}'.format(
+ test_utils.SERVER_GROUP, self.encode_sid, self.encode_did)
+ response = self.tester.post(url)
+ self.assertEquals(response.status_code, 200)
+
+ response_data = json.loads(response.data.decode('utf-8'))
+ self.trans_id = response_data['data']['gridTransId']
+
+ # Check character
+ url = "/sqleditor/query_tool/start/{0}".format(self.trans_id)
+ sql = "select E'{0}';".format(self.test_str)
+ response = self.tester.post(url, data=json.dumps({"sql": sql}),
+ content_type='html/json')
+ self.assertEquals(response.status_code, 200)
+ url = '/sqleditor/poll/{0}'.format(self.trans_id)
+ response = self.tester.get(url)
+ self.assertEquals(response.status_code, 200)
+ response_data = json.loads(response.data.decode('utf-8'))
+ self.assertEquals(response_data['data']['rows_fetched_to'], 1)
+
+ database_utils.disconnect_database(self, self.encode_sid,
+ self.encode_did)
+
+ def tearDown(self):
+ main_conn = test_utils.get_db_connection(
+ self.server['db'],
+ self.server['username'],
+ self.server['db_password'],
+ self.server['host'],
+ self.server['port'],
+ self.server['sslmode']
+ )
+ test_utils.drop_database(main_conn, self.encode_db_name)
diff --git a/web/pgadmin/utils/ajax.py b/web/pgadmin/utils/ajax.py
index 9b44aa3e..c19f778a 100644
--- a/web/pgadmin/utils/ajax.py
+++ b/web/pgadmin/utils/ajax.py
@@ -45,7 +45,8 @@ def get_no_cache_header():
def make_json_response(
- success=1, errormsg='', info='', result=None, data=None, status=200
+ success=1, errormsg='', info='', result=None, data=None, status=200,
+ encoding='utf-8'
):
"""Create a HTML response document describing the results of a request and
containing the data."""
@@ -58,7 +59,7 @@ def make_json_response(
return Response(
response=json.dumps(doc, cls=DataTypeJSONEncoder,
- separators=(',', ':')),
+ separators=(',', ':'), encoding=encoding),
status=status,
mimetype="application/json",
headers=get_no_cache_header()
diff --git a/web/pgadmin/utils/driver/psycopg2/connection.py b/web/pgadmin/utils/driver/psycopg2/connection.py
index cfd161a0..971a24ae 100644
--- a/web/pgadmin/utils/driver/psycopg2/connection.py
+++ b/web/pgadmin/utils/driver/psycopg2/connection.py
@@ -33,7 +33,7 @@ from pgadmin.utils import get_complete_file_path
from ..abstract import BaseDriver, BaseConnection
from .cursor import DictCursor
from .typecast import register_global_typecasters, \
- register_string_typecasters, register_binary_typecasters, \
+ register_binary_typecasters, \
register_array_to_string_typecasters, ALL_JSON_TYPES
@@ -387,8 +387,6 @@ class Connection(BaseConnection):
else:
self.conn.autocommit = True
- register_string_typecasters(self.conn)
-
if self.array_to_string:
register_array_to_string_typecasters(self.conn)
@@ -397,11 +395,22 @@ class Connection(BaseConnection):
if self.use_binary_placeholder:
register_binary_typecasters(self.conn)
- status = _execute(cur, "SET DateStyle=ISO;"
- "SET client_min_messages=notice;"
- "SET bytea_output=escape;"
- "SET client_encoding='UNICODE';")
-
+ if self.conn.encoding not in ('SQL_ASCII', 'SQLASCII',
+ 'MULE_INTERNAL', 'MULEINTERNAL'):
+ status = _execute(cur, "SET DateStyle=ISO;"
+ "SET client_min_messages=notice;"
+ "SET bytea_output=escape;"
+ "SET client_encoding='UNICODE';")
+ self.python_encoding = 'utf-8'
+ encodings[self.conn.encoding] = self.python_encoding
+ else:
+ status = _execute(cur, "SET DateStyle=ISO;"
+ "SET client_min_messages=notice;"
+ "SET bytea_output=escape;"
+ "SET client_encoding='{0}';"
+ .format(self.conn.encoding))
+ self.python_encoding = 'raw_unicode_escape'
+ encodings[self.conn.encoding] = self.python_encoding
if status is not None:
self.conn.close()
self.conn = None
@@ -734,7 +743,7 @@ WHERE
header = []
json_columns = []
- conn_encoding = cur.connection.encoding
+ conn_encoding = encodings[cur.connection.encoding]
for c in cur.ordered_description():
# This is to handle the case in which column name is non-ascii
diff --git a/web/pgadmin/utils/driver/psycopg2/typecast.py b/web/pgadmin/utils/driver/psycopg2/typecast.py
index f1366049..a8f6c385 100644
--- a/web/pgadmin/utils/driver/psycopg2/typecast.py
+++ b/web/pgadmin/utils/driver/psycopg2/typecast.py
@@ -163,49 +163,6 @@ def register_global_typecasters():
psycopg2.extensions.register_type(pg_array_types_to_array_of_string_type)
-def register_string_typecasters(connection):
- if connection.encoding != 'UTF8':
- # In python3 when database encoding is other than utf-8 and client
- # encoding is set to UNICODE then we need to map data from database
- # encoding to utf-8.
- # This is required because when client encoding is set to UNICODE then
- # psycopg assumes database encoding utf-8 and not the actual encoding.
- # Not sure whether it's bug or feature in psycopg for python3.
- if sys.version_info >= (3,):
- def return_as_unicode(value, cursor):
- if value is None:
- return None
- # Treat value as byte sequence of database encoding and then
- # decode it as utf-8 to get correct unicode value.
- return bytes(
- value, encodings[cursor.connection.encoding]
- ).decode('utf-8')
-
- unicode_type = psycopg2.extensions.new_type(
- # "char", name, text, character, character varying
- (19, 18, 25, 1042, 1043, 0),
- 'UNICODE', return_as_unicode)
- else:
- def return_as_unicode(value, cursor):
- if value is None:
- return None
- # Decode it as utf-8 to get correct unicode value.
- return value.decode('utf-8')
-
- unicode_type = psycopg2.extensions.new_type(
- # "char", name, text, character, character varying
- (19, 18, 25, 1042, 1043, 0),
- 'UNICODE', return_as_unicode)
-
- unicode_array_type = psycopg2.extensions.new_array_type(
- # "char"[], name[], text[], character[], character varying[]
- (1002, 1003, 1009, 1014, 1015, 0
- ), 'UNICODEARRAY', unicode_type)
-
- psycopg2.extensions.register_type(unicode_type)
- psycopg2.extensions.register_type(unicode_array_type)
-
-
def register_binary_typecasters(connection):
psycopg2.extensions.register_type(
psycopg2.extensions.new_type(
diff --git a/web/regression/python_test_utils/test_utils.py b/web/regression/python_test_utils/test_utils.py
index 3e517b61..464a09e1 100644
--- a/web/regression/python_test_utils/test_utils.py
+++ b/web/regression/python_test_utils/test_utils.py
@@ -116,7 +116,7 @@ def clear_node_info_dict():
del node_info_dict[node][:]
-def create_database(server, db_name):
+def create_database(server, db_name, encoding=None):
"""This function used to create database and returns the database id"""
try:
connection = get_db_connection(
@@ -130,8 +130,14 @@ def create_database(server, db_name):
old_isolation_level = connection.isolation_level
connection.set_isolation_level(0)
pg_cursor = connection.cursor()
- pg_cursor.execute(
- '''CREATE DATABASE "%s" TEMPLATE template0''' % db_name)
+ if encoding is None:
+ pg_cursor.execute(
+ '''CREATE DATABASE "%s" TEMPLATE template0''' % db_name)
+ else:
+ pg_cursor.execute(
+ '''CREATE DATABASE "%s" TEMPLATE template0
+ ENCODING='%s' LC_COLLATE='%s' LC_CTYPE='%s' ''' %
+ (db_name, encoding[0], encoding[1], encoding[1]))
connection.set_isolation_level(old_isolation_level)
connection.commit()
^ permalink raw reply [nested|flat] 38+ messages in thread
* Re: [pgAdmin4][RM#3289] Can't query SQL_ASCII database.
@ 2018-06-18 20:53 Dave Page <[email protected]>
parent: Aditya Toshniwal <[email protected]>
0 siblings, 1 reply; 38+ messages in thread
From: Dave Page @ 2018-06-18 20:53 UTC (permalink / raw)
To: Aditya Toshniwal <[email protected]>; +Cc: Joao De Almeida Pereira <[email protected]>; pgadmin-hackers
Hi
On Mon, Jun 18, 2018 at 2:14 PM, Aditya Toshniwal <
[email protected]> wrote:
> Hi Hackers,
>
> Attached is the updated patch which includes the fix for Download CSV fail
> in SQL_ASCII database, which is RM3250
> <https://redmine.postgresql.org/issues/3250;
> This should fix RM3289 and RM3250. As they interrelated, sending the
> combined patch.
> Kindly review.
>
This is definitely looking better - both view and save now work as
expected. However, using the test data the I posted upthread, if I try to
edit a value (in this case by adding a couple of chars to the end of the
data in row 2) I get:
2018-06-18 16:41:40,895: SQL pgadmin: Execute (void) for server #1 -
DB:ascii (Query-id: 3093186):
UPDATE public.ascii SET
data = %(data)s::text WHERE
id = '2';
2018-06-18 16:41:41,027: INFO werkzeug: 127.0.0.1 - - [18/Jun/2018
16:41:41] "POST /sqleditor/save/2805058 HTTP/1.1" 500 -
2018-06-18 16:41:41,042: ERROR werkzeug: Error on request:
Traceback (most recent call last):
File
"/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/werkzeug/serving.py",
line 270, in run_wsgi
execute(self.server.app)
File
"/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/werkzeug/serving.py",
line 258, in execute
application_iter = app(environ, start_response)
File
"/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
line 1997, in __call__
return self.wsgi_app(environ, start_response)
File
"/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
line 1985, in wsgi_app
response = self.handle_exception(e)
File
"/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
line 1540, in handle_exception
reraise(exc_type, exc_value, tb)
File
"/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
line 1982, in wsgi_app
response = self.full_dispatch_request()
File
"/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
line 1614, in full_dispatch_request
rv = self.handle_user_exception(e)
File
"/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
line 1517, in handle_user_exception
reraise(exc_type, exc_value, tb)
File
"/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
line 1612, in full_dispatch_request
rv = self.dispatch_request()
File
"/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
line 1598, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File
"/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask_login.py",
line 792, in decorated_view
return func(*args, **kwargs)
File "/Users/dpage/git/pgadmin4/web/pgadmin/tools/sqleditor/__init__.py",
line 776, in save
default_conn)
File "/Users/dpage/git/pgadmin4/web/pgadmin/tools/sqleditor/command.py",
line 829, in save
item['sql'], item['data'])
File
"/Users/dpage/git/pgadmin4/web/pgadmin/utils/driver/psycopg2/connection.py",
line 975, in execute_void
self.__internal_blocking_execute(cur, query, params)
File
"/Users/dpage/git/pgadmin4/web/pgadmin/utils/driver/psycopg2/connection.py",
line 629, in __internal_blocking_execute
cur.execute(query, params)
File
"/Users/dpage/git/pgadmin4/web/pgadmin/utils/driver/psycopg2/cursor.py",
line 176, in execute
return _cursor.execute(self, query, params)
UnicodeEncodeError: 'ascii' codec can't encode characters in position
19-21: ordinal not in range(128)
>
> On Fri, Jun 15, 2018 at 2:33 PM, Aditya Toshniwal <aditya.toshniwal@
> enterprisedb.com> wrote:
>
>> Hi Dave,
>>
>> On Fri, Jun 15, 2018 at 2:08 PM, Dave Page <[email protected]> wrote:
>>
>>> Hi
>>>
>>> On Thu, Jun 14, 2018 at 7:05 AM, Aditya Toshniwal <
>>> [email protected]> wrote:
>>>
>>>> I am sorry I missed the attachment. :(
>>>> PFA.
>>>>
>>>
>>> It looks like the encoding names are getting munged somewhere. I see
>>> you've accounted for that to some degree in connection.py (you have both
>>> SQL_ASCII/MULE_INTERNAL and SQLASCII/MULEINTERNAL), however it doesn't seem
>>> to be enough as I'm getting the following error when trying to download CSV
>>> from the query tool. Can we ensure that conn.encoding contains an un-munged
>>> value at all times, or is that coming from psycopg2?
>>>
>> That is done by pyscopg2 and conn.encoding is a psycopg2 connection
>> property.
>>
>>
>>>
>>> 2018-06-15 09:32:28,799: INFO werkzeug: 127.0.0.1 - - [15/Jun/2018
>>> 09:32:28] "GET /sqleditor/query_tool/download
>>> /2732923?query=SELECT%20*%20FROM%20public.sql_ascii%0AORDER%
>>> 20BY%20id%20ASC%20&filename=sql_ascii.csv HTTP/1.1" 500 -
>>> 2018-06-15 09:32:28,801: ERROR werkzeug: Error on request:
>>> Traceback (most recent call last):
>>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/werkzeug/serving.py",
>>> line 270, in run_wsgi
>>> execute(self.server.app)
>>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/werkzeug/serving.py",
>>> line 260, in execute
>>> for data in application_iter:
>>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/werkzeug/wsgi.py",
>>> line 870, in __next__
>>> return self._next()
>>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/werkzeug/wrappers.py",
>>> line 82, in _iter_encoded
>>> for item in iterable:
>>> File "/Users/dpage/git/pgadmin4/web/pgadmin/utils/driver/psycopg2/connection.py",
>>> line 752, in gen
>>> column_name = column_name.decode(conn_encoding)
>>> LookupError: unknown encoding: SQLASCII
>>>
>>
>> This is because there is code bug here. Below is code used to decode a
>> column name. Connection encoding and python encoding are two different
>> things. Python does not know what SQLASCII is. This will work with UTF-8
>> because python has decoder with same name. I tried to download CSV with the
>> original code without changes and it fails there too. I will fix this and
>> will send the updated patch. I should have checked this.
>> conn_encoding = cur.connection.encoding
>> column_name = column_name.decode(conn_encoding)
>>
>>
>>>
>>> --
>>> Dave Page
>>> Blog: http://pgsnake.blogspot.com
>>> Twitter: @pgsnake
>>>
>>> EnterpriseDB UK: http://www.enterprisedb.com
>>> The Enterprise PostgreSQL Company
>>>
>>
>>
>>
>> --
>> Thanks and Regards,
>> Aditya Toshniwal
>> Software Engineer | EnterpriseDB Software Solutions | Pune
>> "Don't Complain about Heat, Plant a tree"
>>
>
>
>
> --
> Thanks and Regards,
> Aditya Toshniwal
> Software Engineer | EnterpriseDB Software Solutions | Pune
> "Don't Complain about Heat, Plant a tree"
>
--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake
EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 38+ messages in thread
* Re: [pgAdmin4][RM#3289] Can't query SQL_ASCII database.
@ 2018-06-20 14:17 Aditya Toshniwal <[email protected]>
parent: Dave Page <[email protected]>
0 siblings, 1 reply; 38+ messages in thread
From: Aditya Toshniwal @ 2018-06-20 14:17 UTC (permalink / raw)
To: Dave Page <[email protected]>; +Cc: Joao De Almeida Pereira <[email protected]>; pgadmin-hackers
Hi Dave,
Attached is the updated patch. (Playing with encodings is not at all fun :(
)
On Tue, Jun 19, 2018 at 2:23 AM, Dave Page <[email protected]> wrote:
> Hi
>
> On Mon, Jun 18, 2018 at 2:14 PM, Aditya Toshniwal <aditya.toshniwal@
> enterprisedb.com> wrote:
>
>> Hi Hackers,
>>
>> Attached is the updated patch which includes the fix for Download CSV
>> fail in SQL_ASCII database, which is RM3250
>> <https://redmine.postgresql.org/issues/3250;
>> This should fix RM3289 and RM3250. As they interrelated, sending the
>> combined patch.
>> Kindly review.
>>
>
> This is definitely looking better - both view and save now work as
> expected. However, using the test data the I posted upthread, if I try to
> edit a value (in this case by adding a couple of chars to the end of the
> data in row 2) I get:
>
It should fix the error.
>
> 2018-06-18 16:41:40,895: SQL pgadmin: Execute (void) for server #1 -
> DB:ascii (Query-id: 3093186):
> UPDATE public.ascii SET
> data = %(data)s::text WHERE
> id = '2';
> 2018-06-18 16:41:41,027: INFO werkzeug: 127.0.0.1 - - [18/Jun/2018
> 16:41:41] "POST /sqleditor/save/2805058 HTTP/1.1" 500 -
> 2018-06-18 16:41:41,042: ERROR werkzeug: Error on request:
> Traceback (most recent call last):
> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/werkzeug/serving.py",
> line 270, in run_wsgi
> execute(self.server.app)
> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/werkzeug/serving.py",
> line 258, in execute
> application_iter = app(environ, start_response)
> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
> line 1997, in __call__
> return self.wsgi_app(environ, start_response)
> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
> line 1985, in wsgi_app
> response = self.handle_exception(e)
> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
> line 1540, in handle_exception
> reraise(exc_type, exc_value, tb)
> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
> line 1982, in wsgi_app
> response = self.full_dispatch_request()
> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
> line 1614, in full_dispatch_request
> rv = self.handle_user_exception(e)
> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
> line 1517, in handle_user_exception
> reraise(exc_type, exc_value, tb)
> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
> line 1612, in full_dispatch_request
> rv = self.dispatch_request()
> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
> line 1598, in dispatch_request
> return self.view_functions[rule.endpoint](**req.view_args)
> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask_login.py",
> line 792, in decorated_view
> return func(*args, **kwargs)
> File "/Users/dpage/git/pgadmin4/web/pgadmin/tools/sqleditor/__init__.py",
> line 776, in save
> default_conn)
> File "/Users/dpage/git/pgadmin4/web/pgadmin/tools/sqleditor/command.py",
> line 829, in save
> item['sql'], item['data'])
> File "/Users/dpage/git/pgadmin4/web/pgadmin/utils/driver/psycopg2/connection.py",
> line 975, in execute_void
> self.__internal_blocking_execute(cur, query, params)
> File "/Users/dpage/git/pgadmin4/web/pgadmin/utils/driver/psycopg2/connection.py",
> line 629, in __internal_blocking_execute
> cur.execute(query, params)
> File "/Users/dpage/git/pgadmin4/web/pgadmin/utils/driver/psycopg2/cursor.py",
> line 176, in execute
> return _cursor.execute(self, query, params)
> UnicodeEncodeError: 'ascii' codec can't encode characters in position
> 19-21: ordinal not in range(128)
>
>
>>
>> On Fri, Jun 15, 2018 at 2:33 PM, Aditya Toshniwal <
>> [email protected]> wrote:
>>
>>> Hi Dave,
>>>
>>> On Fri, Jun 15, 2018 at 2:08 PM, Dave Page <[email protected]> wrote:
>>>
>>>> Hi
>>>>
>>>> On Thu, Jun 14, 2018 at 7:05 AM, Aditya Toshniwal <
>>>> [email protected]> wrote:
>>>>
>>>>> I am sorry I missed the attachment. :(
>>>>> PFA.
>>>>>
>>>>
>>>> It looks like the encoding names are getting munged somewhere. I see
>>>> you've accounted for that to some degree in connection.py (you have both
>>>> SQL_ASCII/MULE_INTERNAL and SQLASCII/MULEINTERNAL), however it doesn't seem
>>>> to be enough as I'm getting the following error when trying to download CSV
>>>> from the query tool. Can we ensure that conn.encoding contains an un-munged
>>>> value at all times, or is that coming from psycopg2?
>>>>
>>> That is done by pyscopg2 and conn.encoding is a psycopg2 connection
>>> property.
>>>
>>>
>>>>
>>>> 2018-06-15 09:32:28,799: INFO werkzeug: 127.0.0.1 - - [15/Jun/2018
>>>> 09:32:28] "GET /sqleditor/query_tool/download
>>>> /2732923?query=SELECT%20*%20FROM%20public.sql_ascii%0AORDER%
>>>> 20BY%20id%20ASC%20&filename=sql_ascii.csv HTTP/1.1" 500 -
>>>> 2018-06-15 09:32:28,801: ERROR werkzeug: Error on request:
>>>> Traceback (most recent call last):
>>>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/werkzeug/serving.py",
>>>> line 270, in run_wsgi
>>>> execute(self.server.app)
>>>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/werkzeug/serving.py",
>>>> line 260, in execute
>>>> for data in application_iter:
>>>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/werkzeug/wsgi.py",
>>>> line 870, in __next__
>>>> return self._next()
>>>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/werkzeug/wrappers.py",
>>>> line 82, in _iter_encoded
>>>> for item in iterable:
>>>> File "/Users/dpage/git/pgadmin4/web/pgadmin/utils/driver/psycopg2/connection.py",
>>>> line 752, in gen
>>>> column_name = column_name.decode(conn_encoding)
>>>> LookupError: unknown encoding: SQLASCII
>>>>
>>>
>>> This is because there is code bug here. Below is code used to decode a
>>> column name. Connection encoding and python encoding are two different
>>> things. Python does not know what SQLASCII is. This will work with UTF-8
>>> because python has decoder with same name. I tried to download CSV with the
>>> original code without changes and it fails there too. I will fix this and
>>> will send the updated patch. I should have checked this.
>>> conn_encoding = cur.connection.encoding
>>> column_name = column_name.decode(conn_encoding)
>>>
>>>
>>>>
>>>> --
>>>> Dave Page
>>>> Blog: http://pgsnake.blogspot.com
>>>> Twitter: @pgsnake
>>>>
>>>> EnterpriseDB UK: http://www.enterprisedb.com
>>>> The Enterprise PostgreSQL Company
>>>>
>>>
>>>
>>>
>>> --
>>> Thanks and Regards,
>>> Aditya Toshniwal
>>> Software Engineer | EnterpriseDB Software Solutions | Pune
>>> "Don't Complain about Heat, Plant a tree"
>>>
>>
>>
>>
>> --
>> Thanks and Regards,
>> Aditya Toshniwal
>> Software Engineer | EnterpriseDB Software Solutions | Pune
>> "Don't Complain about Heat, Plant a tree"
>>
>
>
>
> --
> Dave Page
> Blog: http://pgsnake.blogspot.com
> Twitter: @pgsnake
>
> EnterpriseDB UK: http://www.enterprisedb.com
> The Enterprise PostgreSQL Company
>
--
Thanks and Regards,
Aditya Toshniwal
Software Engineer | EnterpriseDB Software Solutions | Pune
"Don't Complain about Heat, Plant a tree"
Attachments:
[application/octet-stream] RM3289_3250.patch (14.3K, 3-RM3289_3250.patch)
download | inline diff:
diff --git a/web/pgadmin/tools/sqleditor/__init__.py b/web/pgadmin/tools/sqleditor/__init__.py
index a9460dd7..d8f2ce67 100644
--- a/web/pgadmin/tools/sqleditor/__init__.py
+++ b/web/pgadmin/tools/sqleditor/__init__.py
@@ -574,7 +574,8 @@ def poll(trans_id):
'client_primary_key': client_primary_key,
'has_oids': has_oids,
'oids': oids
- }
+ },
+ encoding=conn.python_encoding
)
@@ -646,7 +647,8 @@ def fetch(trans_id, fetch_all=None):
'has_more_rows': has_more_rows,
'rows_fetched_from': rows_fetched_from,
'rows_fetched_to': rows_fetched_to
- }
+ },
+ encoding=conn.python_encoding
)
diff --git a/web/pgadmin/tools/sqleditor/tests/test_encoding_charset.py b/web/pgadmin/tools/sqleditor/tests/test_encoding_charset.py
new file mode 100644
index 00000000..8a8d45e3
--- /dev/null
+++ b/web/pgadmin/tools/sqleditor/tests/test_encoding_charset.py
@@ -0,0 +1,113 @@
+##########################################################################
+#
+# pgAdmin 4 - PostgreSQL Tools
+#
+# Copyright (C) 2013 - 2018, The pgAdmin Development Team
+# This software is released under the PostgreSQL Licence
+#
+##########################################################################
+
+from pgadmin.utils.route import BaseTestGenerator
+from pgadmin.browser.server_groups.servers.databases.tests import utils as \
+ database_utils
+from regression import parent_node_dict
+from regression.python_test_utils import test_utils
+import json
+from pgadmin.utils import server_utils
+
+
+class TestEncodingCharset(BaseTestGenerator):
+ """
+ This class validates character support in pgAdmin4 for
+ different PostgresDB encodings
+ """
+ skip_on_database = ['gpdb']
+ scenarios = [
+ (
+ 'With Encoding UTF8',
+ dict(
+ db_encoding='UTF8',
+ lc_collate='C',
+ test_str='A'
+ )),
+ (
+ 'With Encoding WIN1252',
+ dict(
+ db_encoding='WIN1252',
+ lc_collate='C',
+ test_str='A'
+ )),
+ (
+ 'With Encoding EUC_CN',
+ dict(
+ db_encoding='EUC_CN',
+ lc_collate='C',
+ test_str='A'
+ )),
+ (
+ 'With Encoding SQL_ASCII',
+ dict(
+ db_encoding='SQL_ASCII',
+ lc_collate='C',
+ test_str='\\255'
+ )),
+ ]
+
+ def setUp(self):
+ self.encode_db_name = 'encoding_' + self.db_encoding
+ self.encode_sid = self.server_information['server_id']
+
+ server_con = server_utils.connect_server(self, self.encode_sid)
+ if hasattr(self, 'skip_on_database'):
+ if 'data' in server_con and 'type' in server_con['data']:
+ if server_con['data']['type'] in self.skip_on_database:
+ self.skipTest('cannot run in: %s' %
+ server_con['data']['type'])
+
+ self.encode_did = test_utils.create_database(
+ self.server, self.encode_db_name,
+ (self.db_encoding, self.lc_collate))
+
+ def runTest(self):
+
+ db_con = database_utils.connect_database(self,
+ test_utils.SERVER_GROUP,
+ self.encode_sid,
+ self.encode_did)
+ if not db_con["info"] == "Database connected.":
+ raise Exception("Could not connect to the database.")
+
+ # Initialize query tool
+ url = '/datagrid/initialize/query_tool/{0}/{1}/{2}'.format(
+ test_utils.SERVER_GROUP, self.encode_sid, self.encode_did)
+ response = self.tester.post(url)
+ self.assertEquals(response.status_code, 200)
+
+ response_data = json.loads(response.data.decode('utf-8'))
+ self.trans_id = response_data['data']['gridTransId']
+
+ # Check character
+ url = "/sqleditor/query_tool/start/{0}".format(self.trans_id)
+ sql = "select E'{0}';".format(self.test_str)
+ response = self.tester.post(url, data=json.dumps({"sql": sql}),
+ content_type='html/json')
+ self.assertEquals(response.status_code, 200)
+ url = '/sqleditor/poll/{0}'.format(self.trans_id)
+ response = self.tester.get(url)
+ self.assertEquals(response.status_code, 200)
+ response_data = json.loads(response.data.decode('utf-8'))
+ self.assertEquals(response_data['data']['rows_fetched_to'], 1)
+
+ database_utils.disconnect_database(self, self.encode_sid,
+ self.encode_did)
+
+ def tearDown(self):
+ main_conn = test_utils.get_db_connection(
+ self.server['db'],
+ self.server['username'],
+ self.server['db_password'],
+ self.server['host'],
+ self.server['port'],
+ self.server['sslmode']
+ )
+ test_utils.drop_database(main_conn, self.encode_db_name)
diff --git a/web/pgadmin/utils/ajax.py b/web/pgadmin/utils/ajax.py
index 9b44aa3e..c19f778a 100644
--- a/web/pgadmin/utils/ajax.py
+++ b/web/pgadmin/utils/ajax.py
@@ -45,7 +45,8 @@ def get_no_cache_header():
def make_json_response(
- success=1, errormsg='', info='', result=None, data=None, status=200
+ success=1, errormsg='', info='', result=None, data=None, status=200,
+ encoding='utf-8'
):
"""Create a HTML response document describing the results of a request and
containing the data."""
@@ -58,7 +59,7 @@ def make_json_response(
return Response(
response=json.dumps(doc, cls=DataTypeJSONEncoder,
- separators=(',', ':')),
+ separators=(',', ':'), encoding=encoding),
status=status,
mimetype="application/json",
headers=get_no_cache_header()
diff --git a/web/pgadmin/utils/driver/psycopg2/connection.py b/web/pgadmin/utils/driver/psycopg2/connection.py
index cfd161a0..6f522da6 100644
--- a/web/pgadmin/utils/driver/psycopg2/connection.py
+++ b/web/pgadmin/utils/driver/psycopg2/connection.py
@@ -50,7 +50,6 @@ else:
_ = gettext
-
# Register global type caster which will be applicable to all connections.
register_global_typecasters()
@@ -397,10 +396,29 @@ class Connection(BaseConnection):
if self.use_binary_placeholder:
register_binary_typecasters(self.conn)
- status = _execute(cur, "SET DateStyle=ISO;"
- "SET client_min_messages=notice;"
- "SET bytea_output=escape;"
- "SET client_encoding='UNICODE';")
+ if self.conn.encoding in ('SQL_ASCII', 'SQLASCII',
+ 'MULE_INTERNAL', 'MULEINTERNAL'):
+ status = _execute(cur, "SET DateStyle=ISO;"
+ "SET client_min_messages=notice;"
+ "SET bytea_output=escape;"
+ "SET client_encoding='{0}';"
+ .format(self.conn.encoding))
+ self.python_encoding = 'raw_unicode_escape'
+ else:
+ status = _execute(cur, "SET DateStyle=ISO;"
+ "SET client_min_messages=notice;"
+ "SET bytea_output=escape;"
+ "SET client_encoding='UNICODE';")
+ self.python_encoding = 'utf-8'
+
+ # Replace the python encoding for original name and renamed encodings
+ # psycopg2 removes the underscore in conn.encoding
+ # Setting the encodings dict value will only help for select statements
+ # because for parameterized DML, param values are converted based on
+ # python encoding of pyscopg2s internal encodings dict.
+ for key, val in encodings.items():
+ if key.replace('_', '') == self.conn.encoding:
+ encodings[key] = self.python_encoding
if status is not None:
self.conn.close()
@@ -598,6 +616,21 @@ WHERE
return True, cur
+ def escape_params_sqlascii(self, params):
+ # The data is unescaped using string_typecasters when selected
+ # We need to esacpe the data so that it does not fail when
+ # it is encoded with python ascii
+ # unicode_escape helps in escaping and unescaping
+ if self.conn.encoding in ('SQL_ASCII', 'SQLASCII',
+ 'MULE_INTERNAL', 'MULEINTERNAL')\
+ and params is not None and type(params) == dict:
+ params = {
+ key: val.encode('unicode_escape')
+ .decode('raw_unicode_escape')
+ for key, val in params.items()
+ }
+ return params
+
def __internal_blocking_execute(self, cur, query, params):
"""
This function executes the query using cursor's execute function,
@@ -617,6 +650,7 @@ WHERE
else:
query = query.encode('utf-8')
+ params = self.escape_params_sqlascii(params)
cur.execute(query, params)
if self.async == 1:
self._wait(cur.connection)
@@ -734,7 +768,7 @@ WHERE
header = []
json_columns = []
- conn_encoding = cur.connection.encoding
+ conn_encoding = encodings[cur.connection.encoding]
for c in cur.ordered_description():
# This is to handle the case in which column name is non-ascii
@@ -880,6 +914,9 @@ WHERE
else:
query = query.encode('utf-8')
+ # Convert the params based on python_encoding
+ params = self.escape_params_sqlascii(params)
+
self.__async_cursor = None
status, cur = self.__cursor()
diff --git a/web/pgadmin/utils/driver/psycopg2/typecast.py b/web/pgadmin/utils/driver/psycopg2/typecast.py
index f1366049..de2e6325 100644
--- a/web/pgadmin/utils/driver/psycopg2/typecast.py
+++ b/web/pgadmin/utils/driver/psycopg2/typecast.py
@@ -164,46 +164,37 @@ def register_global_typecasters():
def register_string_typecasters(connection):
- if connection.encoding != 'UTF8':
- # In python3 when database encoding is other than utf-8 and client
- # encoding is set to UNICODE then we need to map data from database
- # encoding to utf-8.
- # This is required because when client encoding is set to UNICODE then
- # psycopg assumes database encoding utf-8 and not the actual encoding.
- # Not sure whether it's bug or feature in psycopg for python3.
+ # raw_unicode_escape used for SQL ASCII will escape the
+ # characters. Here we unescape them using unicode_escape
+ # and send ahead. When insert update is done, the characters
+ # are escaped again and sent to the DB.
+ if connection.encoding in ('SQL_ASCII', 'SQLASCII',
+ 'MULE_INTERNAL', 'MULEINTERNAL'):
if sys.version_info >= (3,):
- def return_as_unicode(value, cursor):
+ def non_ascii_escape(value, cursor):
if value is None:
return None
- # Treat value as byte sequence of database encoding and then
- # decode it as utf-8 to get correct unicode value.
return bytes(
value, encodings[cursor.connection.encoding]
- ).decode('utf-8')
-
- unicode_type = psycopg2.extensions.new_type(
- # "char", name, text, character, character varying
- (19, 18, 25, 1042, 1043, 0),
- 'UNICODE', return_as_unicode)
+ ).decode('unicode_escape')
else:
- def return_as_unicode(value, cursor):
+ def non_ascii_escape(value, cursor):
if value is None:
return None
- # Decode it as utf-8 to get correct unicode value.
- return value.decode('utf-8')
+ return value.decode('unicode_escape')
- unicode_type = psycopg2.extensions.new_type(
- # "char", name, text, character, character varying
- (19, 18, 25, 1042, 1043, 0),
- 'UNICODE', return_as_unicode)
+ unicode_type = psycopg2.extensions.new_type(
+ # "char", name, text, character, character varying
+ (19, 18, 25, 1042, 1043, 0),
+ 'UNICODE', non_ascii_escape)
unicode_array_type = psycopg2.extensions.new_array_type(
# "char"[], name[], text[], character[], character varying[]
(1002, 1003, 1009, 1014, 1015, 0
), 'UNICODEARRAY', unicode_type)
- psycopg2.extensions.register_type(unicode_type)
- psycopg2.extensions.register_type(unicode_array_type)
+ psycopg2.extensions.register_type(unicode_type, connection)
+ psycopg2.extensions.register_type(unicode_array_type, connection)
def register_binary_typecasters(connection):
diff --git a/web/regression/python_test_utils/test_utils.py b/web/regression/python_test_utils/test_utils.py
index 3e517b61..464a09e1 100644
--- a/web/regression/python_test_utils/test_utils.py
+++ b/web/regression/python_test_utils/test_utils.py
@@ -116,7 +116,7 @@ def clear_node_info_dict():
del node_info_dict[node][:]
-def create_database(server, db_name):
+def create_database(server, db_name, encoding=None):
"""This function used to create database and returns the database id"""
try:
connection = get_db_connection(
@@ -130,8 +130,14 @@ def create_database(server, db_name):
old_isolation_level = connection.isolation_level
connection.set_isolation_level(0)
pg_cursor = connection.cursor()
- pg_cursor.execute(
- '''CREATE DATABASE "%s" TEMPLATE template0''' % db_name)
+ if encoding is None:
+ pg_cursor.execute(
+ '''CREATE DATABASE "%s" TEMPLATE template0''' % db_name)
+ else:
+ pg_cursor.execute(
+ '''CREATE DATABASE "%s" TEMPLATE template0
+ ENCODING='%s' LC_COLLATE='%s' LC_CTYPE='%s' ''' %
+ (db_name, encoding[0], encoding[1], encoding[1]))
connection.set_isolation_level(old_isolation_level)
connection.commit()
^ permalink raw reply [nested|flat] 38+ messages in thread
* Re: [pgAdmin4][RM#3289] Can't query SQL_ASCII database.
@ 2018-06-21 12:57 Dave Page <[email protected]>
parent: Aditya Toshniwal <[email protected]>
0 siblings, 1 reply; 38+ messages in thread
From: Dave Page @ 2018-06-21 12:57 UTC (permalink / raw)
To: Aditya Toshniwal <[email protected]>; +Cc: Joao De Almeida Pereira <[email protected]>; pgadmin-hackers
Thanks - patch applied!
On Wed, Jun 20, 2018 at 3:17 PM, Aditya Toshniwal <
[email protected]> wrote:
> Hi Dave,
>
> Attached is the updated patch. (Playing with encodings is not at all fun
> :( )
>
> On Tue, Jun 19, 2018 at 2:23 AM, Dave Page <[email protected]> wrote:
>
>> Hi
>>
>> On Mon, Jun 18, 2018 at 2:14 PM, Aditya Toshniwal <
>> [email protected]> wrote:
>>
>>> Hi Hackers,
>>>
>>> Attached is the updated patch which includes the fix for Download CSV
>>> fail in SQL_ASCII database, which is RM3250
>>> <https://redmine.postgresql.org/issues/3250;
>>> This should fix RM3289 and RM3250. As they interrelated, sending the
>>> combined patch.
>>> Kindly review.
>>>
>>
>> This is definitely looking better - both view and save now work as
>> expected. However, using the test data the I posted upthread, if I try to
>> edit a value (in this case by adding a couple of chars to the end of the
>> data in row 2) I get:
>>
> It should fix the error.
>
>>
>> 2018-06-18 16:41:40,895: SQL pgadmin: Execute (void) for server #1 -
>> DB:ascii (Query-id: 3093186):
>> UPDATE public.ascii SET
>> data = %(data)s::text WHERE
>> id = '2';
>> 2018-06-18 16:41:41,027: INFO werkzeug: 127.0.0.1 - - [18/Jun/2018
>> 16:41:41] "POST /sqleditor/save/2805058 HTTP/1.1" 500 -
>> 2018-06-18 16:41:41,042: ERROR werkzeug: Error on request:
>> Traceback (most recent call last):
>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/werkzeug/serving.py",
>> line 270, in run_wsgi
>> execute(self.server.app)
>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/werkzeug/serving.py",
>> line 258, in execute
>> application_iter = app(environ, start_response)
>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
>> line 1997, in __call__
>> return self.wsgi_app(environ, start_response)
>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
>> line 1985, in wsgi_app
>> response = self.handle_exception(e)
>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
>> line 1540, in handle_exception
>> reraise(exc_type, exc_value, tb)
>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
>> line 1982, in wsgi_app
>> response = self.full_dispatch_request()
>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
>> line 1614, in full_dispatch_request
>> rv = self.handle_user_exception(e)
>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
>> line 1517, in handle_user_exception
>> reraise(exc_type, exc_value, tb)
>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
>> line 1612, in full_dispatch_request
>> rv = self.dispatch_request()
>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
>> line 1598, in dispatch_request
>> return self.view_functions[rule.endpoint](**req.view_args)
>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask_login.py",
>> line 792, in decorated_view
>> return func(*args, **kwargs)
>> File "/Users/dpage/git/pgadmin4/web/pgadmin/tools/sqleditor/__init__.py",
>> line 776, in save
>> default_conn)
>> File "/Users/dpage/git/pgadmin4/web/pgadmin/tools/sqleditor/command.py",
>> line 829, in save
>> item['sql'], item['data'])
>> File "/Users/dpage/git/pgadmin4/web/pgadmin/utils/driver/psycopg2/connection.py",
>> line 975, in execute_void
>> self.__internal_blocking_execute(cur, query, params)
>> File "/Users/dpage/git/pgadmin4/web/pgadmin/utils/driver/psycopg2/connection.py",
>> line 629, in __internal_blocking_execute
>> cur.execute(query, params)
>> File "/Users/dpage/git/pgadmin4/web/pgadmin/utils/driver/psycopg2/cursor.py",
>> line 176, in execute
>> return _cursor.execute(self, query, params)
>> UnicodeEncodeError: 'ascii' codec can't encode characters in position
>> 19-21: ordinal not in range(128)
>>
>>
>>>
>>> On Fri, Jun 15, 2018 at 2:33 PM, Aditya Toshniwal <
>>> [email protected]> wrote:
>>>
>>>> Hi Dave,
>>>>
>>>> On Fri, Jun 15, 2018 at 2:08 PM, Dave Page <[email protected]> wrote:
>>>>
>>>>> Hi
>>>>>
>>>>> On Thu, Jun 14, 2018 at 7:05 AM, Aditya Toshniwal <
>>>>> [email protected]> wrote:
>>>>>
>>>>>> I am sorry I missed the attachment. :(
>>>>>> PFA.
>>>>>>
>>>>>
>>>>> It looks like the encoding names are getting munged somewhere. I see
>>>>> you've accounted for that to some degree in connection.py (you have both
>>>>> SQL_ASCII/MULE_INTERNAL and SQLASCII/MULEINTERNAL), however it doesn't seem
>>>>> to be enough as I'm getting the following error when trying to download CSV
>>>>> from the query tool. Can we ensure that conn.encoding contains an un-munged
>>>>> value at all times, or is that coming from psycopg2?
>>>>>
>>>> That is done by pyscopg2 and conn.encoding is a psycopg2 connection
>>>> property.
>>>>
>>>>
>>>>>
>>>>> 2018-06-15 09:32:28,799: INFO werkzeug: 127.0.0.1 - - [15/Jun/2018
>>>>> 09:32:28] "GET /sqleditor/query_tool/download
>>>>> /2732923?query=SELECT%20*%20FROM%20public.sql_ascii%0AORDER%
>>>>> 20BY%20id%20ASC%20&filename=sql_ascii.csv HTTP/1.1" 500 -
>>>>> 2018-06-15 09:32:28,801: ERROR werkzeug: Error on request:
>>>>> Traceback (most recent call last):
>>>>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/werkzeug/serving.py",
>>>>> line 270, in run_wsgi
>>>>> execute(self.server.app)
>>>>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/werkzeug/serving.py",
>>>>> line 260, in execute
>>>>> for data in application_iter:
>>>>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/werkzeug/wsgi.py",
>>>>> line 870, in __next__
>>>>> return self._next()
>>>>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/werkzeug/wrappers.py",
>>>>> line 82, in _iter_encoded
>>>>> for item in iterable:
>>>>> File "/Users/dpage/git/pgadmin4/web/pgadmin/utils/driver/psycopg2/connection.py",
>>>>> line 752, in gen
>>>>> column_name = column_name.decode(conn_encoding)
>>>>> LookupError: unknown encoding: SQLASCII
>>>>>
>>>>
>>>> This is because there is code bug here. Below is code used to decode a
>>>> column name. Connection encoding and python encoding are two different
>>>> things. Python does not know what SQLASCII is. This will work with UTF-8
>>>> because python has decoder with same name. I tried to download CSV with the
>>>> original code without changes and it fails there too. I will fix this and
>>>> will send the updated patch. I should have checked this.
>>>> conn_encoding = cur.connection.encoding
>>>> column_name = column_name.decode(conn_encoding)
>>>>
>>>>
>>>>>
>>>>> --
>>>>> Dave Page
>>>>> Blog: http://pgsnake.blogspot.com
>>>>> Twitter: @pgsnake
>>>>>
>>>>> EnterpriseDB UK: http://www.enterprisedb.com
>>>>> The Enterprise PostgreSQL Company
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Thanks and Regards,
>>>> Aditya Toshniwal
>>>> Software Engineer | EnterpriseDB Software Solutions | Pune
>>>> "Don't Complain about Heat, Plant a tree"
>>>>
>>>
>>>
>>>
>>> --
>>> Thanks and Regards,
>>> Aditya Toshniwal
>>> Software Engineer | EnterpriseDB Software Solutions | Pune
>>> "Don't Complain about Heat, Plant a tree"
>>>
>>
>>
>>
>> --
>> Dave Page
>> Blog: http://pgsnake.blogspot.com
>> Twitter: @pgsnake
>>
>> EnterpriseDB UK: http://www.enterprisedb.com
>> The Enterprise PostgreSQL Company
>>
>
>
>
> --
> Thanks and Regards,
> Aditya Toshniwal
> Software Engineer | EnterpriseDB Software Solutions | Pune
> "Don't Complain about Heat, Plant a tree"
>
--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake
EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 38+ messages in thread
* Re: [pgAdmin4][RM#3289] Can't query SQL_ASCII database.
@ 2018-06-21 15:51 Aditya Toshniwal <[email protected]>
parent: Dave Page <[email protected]>
0 siblings, 1 reply; 38+ messages in thread
From: Aditya Toshniwal @ 2018-06-21 15:51 UTC (permalink / raw)
To: Dave Page <[email protected]>; +Cc: Joao De Almeida Pereira <[email protected]>; pgadmin-hackers
Hi Hackers,
PFA patch to make SQL ASCII related changes compatible with Python 2.6.
Dictionary comprehension is not supported in Python 2.6.
On Thu, Jun 21, 2018 at 6:27 PM, Dave Page <[email protected]> wrote:
> Thanks - patch applied!
>
> On Wed, Jun 20, 2018 at 3:17 PM, Aditya Toshniwal <aditya.toshniwal@
> enterprisedb.com> wrote:
>
>> Hi Dave,
>>
>> Attached is the updated patch. (Playing with encodings is not at all fun
>> :( )
>>
>> On Tue, Jun 19, 2018 at 2:23 AM, Dave Page <[email protected]> wrote:
>>
>>> Hi
>>>
>>> On Mon, Jun 18, 2018 at 2:14 PM, Aditya Toshniwal <
>>> [email protected]> wrote:
>>>
>>>> Hi Hackers,
>>>>
>>>> Attached is the updated patch which includes the fix for Download CSV
>>>> fail in SQL_ASCII database, which is RM3250
>>>> <https://redmine.postgresql.org/issues/3250;
>>>> This should fix RM3289 and RM3250. As they interrelated, sending the
>>>> combined patch.
>>>> Kindly review.
>>>>
>>>
>>> This is definitely looking better - both view and save now work as
>>> expected. However, using the test data the I posted upthread, if I try to
>>> edit a value (in this case by adding a couple of chars to the end of the
>>> data in row 2) I get:
>>>
>> It should fix the error.
>>
>>>
>>> 2018-06-18 16:41:40,895: SQL pgadmin: Execute (void) for server #1 -
>>> DB:ascii (Query-id: 3093186):
>>> UPDATE public.ascii SET
>>> data = %(data)s::text WHERE
>>> id = '2';
>>> 2018-06-18 16:41:41,027: INFO werkzeug: 127.0.0.1 - - [18/Jun/2018
>>> 16:41:41] "POST /sqleditor/save/2805058 HTTP/1.1" 500 -
>>> 2018-06-18 16:41:41,042: ERROR werkzeug: Error on request:
>>> Traceback (most recent call last):
>>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/werkzeug/serving.py",
>>> line 270, in run_wsgi
>>> execute(self.server.app)
>>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/werkzeug/serving.py",
>>> line 258, in execute
>>> application_iter = app(environ, start_response)
>>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
>>> line 1997, in __call__
>>> return self.wsgi_app(environ, start_response)
>>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
>>> line 1985, in wsgi_app
>>> response = self.handle_exception(e)
>>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
>>> line 1540, in handle_exception
>>> reraise(exc_type, exc_value, tb)
>>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
>>> line 1982, in wsgi_app
>>> response = self.full_dispatch_request()
>>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
>>> line 1614, in full_dispatch_request
>>> rv = self.handle_user_exception(e)
>>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
>>> line 1517, in handle_user_exception
>>> reraise(exc_type, exc_value, tb)
>>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
>>> line 1612, in full_dispatch_request
>>> rv = self.dispatch_request()
>>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
>>> line 1598, in dispatch_request
>>> return self.view_functions[rule.endpoint](**req.view_args)
>>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask_login.py",
>>> line 792, in decorated_view
>>> return func(*args, **kwargs)
>>> File "/Users/dpage/git/pgadmin4/web/pgadmin/tools/sqleditor/__init__.py",
>>> line 776, in save
>>> default_conn)
>>> File "/Users/dpage/git/pgadmin4/web/pgadmin/tools/sqleditor/command.py",
>>> line 829, in save
>>> item['sql'], item['data'])
>>> File "/Users/dpage/git/pgadmin4/web/pgadmin/utils/driver/psycopg2/connection.py",
>>> line 975, in execute_void
>>> self.__internal_blocking_execute(cur, query, params)
>>> File "/Users/dpage/git/pgadmin4/web/pgadmin/utils/driver/psycopg2/connection.py",
>>> line 629, in __internal_blocking_execute
>>> cur.execute(query, params)
>>> File "/Users/dpage/git/pgadmin4/web/pgadmin/utils/driver/psycopg2/cursor.py",
>>> line 176, in execute
>>> return _cursor.execute(self, query, params)
>>> UnicodeEncodeError: 'ascii' codec can't encode characters in position
>>> 19-21: ordinal not in range(128)
>>>
>>>
>>>>
>>>> On Fri, Jun 15, 2018 at 2:33 PM, Aditya Toshniwal <
>>>> [email protected]> wrote:
>>>>
>>>>> Hi Dave,
>>>>>
>>>>> On Fri, Jun 15, 2018 at 2:08 PM, Dave Page <[email protected]> wrote:
>>>>>
>>>>>> Hi
>>>>>>
>>>>>> On Thu, Jun 14, 2018 at 7:05 AM, Aditya Toshniwal <
>>>>>> [email protected]> wrote:
>>>>>>
>>>>>>> I am sorry I missed the attachment. :(
>>>>>>> PFA.
>>>>>>>
>>>>>>
>>>>>> It looks like the encoding names are getting munged somewhere. I see
>>>>>> you've accounted for that to some degree in connection.py (you have both
>>>>>> SQL_ASCII/MULE_INTERNAL and SQLASCII/MULEINTERNAL), however it doesn't seem
>>>>>> to be enough as I'm getting the following error when trying to download CSV
>>>>>> from the query tool. Can we ensure that conn.encoding contains an un-munged
>>>>>> value at all times, or is that coming from psycopg2?
>>>>>>
>>>>> That is done by pyscopg2 and conn.encoding is a psycopg2 connection
>>>>> property.
>>>>>
>>>>>
>>>>>>
>>>>>> 2018-06-15 09:32:28,799: INFO werkzeug: 127.0.0.1 - - [15/Jun/2018
>>>>>> 09:32:28] "GET /sqleditor/query_tool/download
>>>>>> /2732923?query=SELECT%20*%20FROM%20public.sql_ascii%0AORDER%
>>>>>> 20BY%20id%20ASC%20&filename=sql_ascii.csv HTTP/1.1" 500 -
>>>>>> 2018-06-15 09:32:28,801: ERROR werkzeug: Error on request:
>>>>>> Traceback (most recent call last):
>>>>>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/werkzeug/serving.py",
>>>>>> line 270, in run_wsgi
>>>>>> execute(self.server.app)
>>>>>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/werkzeug/serving.py",
>>>>>> line 260, in execute
>>>>>> for data in application_iter:
>>>>>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/werkzeug/wsgi.py",
>>>>>> line 870, in __next__
>>>>>> return self._next()
>>>>>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/werkzeug/wrappers.py",
>>>>>> line 82, in _iter_encoded
>>>>>> for item in iterable:
>>>>>> File "/Users/dpage/git/pgadmin4/web/pgadmin/utils/driver/psycopg2/connection.py",
>>>>>> line 752, in gen
>>>>>> column_name = column_name.decode(conn_encoding)
>>>>>> LookupError: unknown encoding: SQLASCII
>>>>>>
>>>>>
>>>>> This is because there is code bug here. Below is code used to decode
>>>>> a column name. Connection encoding and python encoding are two different
>>>>> things. Python does not know what SQLASCII is. This will work with UTF-8
>>>>> because python has decoder with same name. I tried to download CSV with the
>>>>> original code without changes and it fails there too. I will fix this and
>>>>> will send the updated patch. I should have checked this.
>>>>> conn_encoding = cur.connection.encoding
>>>>> column_name = column_name.decode(conn_encoding)
>>>>>
>>>>>
>>>>>>
>>>>>> --
>>>>>> Dave Page
>>>>>> Blog: http://pgsnake.blogspot.com
>>>>>> Twitter: @pgsnake
>>>>>>
>>>>>> EnterpriseDB UK: http://www.enterprisedb.com
>>>>>> The Enterprise PostgreSQL Company
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Thanks and Regards,
>>>>> Aditya Toshniwal
>>>>> Software Engineer | EnterpriseDB Software Solutions | Pune
>>>>> "Don't Complain about Heat, Plant a tree"
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Thanks and Regards,
>>>> Aditya Toshniwal
>>>> Software Engineer | EnterpriseDB Software Solutions | Pune
>>>> "Don't Complain about Heat, Plant a tree"
>>>>
>>>
>>>
>>>
>>> --
>>> Dave Page
>>> Blog: http://pgsnake.blogspot.com
>>> Twitter: @pgsnake
>>>
>>> EnterpriseDB UK: http://www.enterprisedb.com
>>> The Enterprise PostgreSQL Company
>>>
>>
>>
>>
>> --
>> Thanks and Regards,
>> Aditya Toshniwal
>> Software Engineer | EnterpriseDB Software Solutions | Pune
>> "Don't Complain about Heat, Plant a tree"
>>
>
>
>
> --
> Dave Page
> Blog: http://pgsnake.blogspot.com
> Twitter: @pgsnake
>
> EnterpriseDB UK: http://www.enterprisedb.com
> The Enterprise PostgreSQL Company
>
--
Thanks and Regards,
Aditya Toshniwal
Software Engineer | EnterpriseDB Software Solutions | Pune
"Don't Complain about Heat, Plant a tree"
Attachments:
[application/octet-stream] RM3289_py26.patch (1.4K, 3-RM3289_py26.patch)
download | inline diff:
diff --git a/web/pgadmin/utils/driver/psycopg2/connection.py b/web/pgadmin/utils/driver/psycopg2/connection.py
index e8ed88a3..1b4d2a3f 100644
--- a/web/pgadmin/utils/driver/psycopg2/connection.py
+++ b/web/pgadmin/utils/driver/psycopg2/connection.py
@@ -622,14 +622,15 @@ WHERE
# We need to esacpe the data so that it does not fail when
# it is encoded with python ascii
# unicode_escape helps in escaping and unescaping
- if self.conn.encoding in ('SQL_ASCII', 'SQLASCII',
- 'MULE_INTERNAL', 'MULEINTERNAL')\
- and params is not None and type(params) == dict:
- params = {
- key: val.encode('unicode_escape')
- .decode('raw_unicode_escape')
- for key, val in params.items()
- }
+ if self.conn:
+ if self.conn.encoding in ('SQL_ASCII', 'SQLASCII',
+ 'MULE_INTERNAL', 'MULEINTERNAL')\
+ and params is not None and type(params) == dict:
+ params = dict(
+ (key, val.encode('unicode_escape')
+ .decode('raw_unicode_escape'))
+ for key, val in params.items()
+ )
return params
def __internal_blocking_execute(self, cur, query, params):
^ permalink raw reply [nested|flat] 38+ messages in thread
* Re: [pgAdmin4][RM#3289] Can't query SQL_ASCII database.
@ 2018-06-21 21:07 Dave Page <[email protected]>
parent: Aditya Toshniwal <[email protected]>
0 siblings, 0 replies; 38+ messages in thread
From: Dave Page @ 2018-06-21 21:07 UTC (permalink / raw)
To: Aditya Toshniwal <[email protected]>; +Cc: Joao De Almeida Pereira <[email protected]>; pgadmin-hackers
Thanks, applied.
On Thu, Jun 21, 2018 at 4:51 PM, Aditya Toshniwal <
[email protected]> wrote:
> Hi Hackers,
>
> PFA patch to make SQL ASCII related changes compatible with Python 2.6.
> Dictionary comprehension is not supported in Python 2.6.
>
> On Thu, Jun 21, 2018 at 6:27 PM, Dave Page <[email protected]> wrote:
>
>> Thanks - patch applied!
>>
>> On Wed, Jun 20, 2018 at 3:17 PM, Aditya Toshniwal <
>> [email protected]> wrote:
>>
>>> Hi Dave,
>>>
>>> Attached is the updated patch. (Playing with encodings is not at all fun
>>> :( )
>>>
>>> On Tue, Jun 19, 2018 at 2:23 AM, Dave Page <[email protected]> wrote:
>>>
>>>> Hi
>>>>
>>>> On Mon, Jun 18, 2018 at 2:14 PM, Aditya Toshniwal <
>>>> [email protected]> wrote:
>>>>
>>>>> Hi Hackers,
>>>>>
>>>>> Attached is the updated patch which includes the fix for Download CSV
>>>>> fail in SQL_ASCII database, which is RM3250
>>>>> <https://redmine.postgresql.org/issues/3250;
>>>>> This should fix RM3289 and RM3250. As they interrelated, sending the
>>>>> combined patch.
>>>>> Kindly review.
>>>>>
>>>>
>>>> This is definitely looking better - both view and save now work as
>>>> expected. However, using the test data the I posted upthread, if I try to
>>>> edit a value (in this case by adding a couple of chars to the end of the
>>>> data in row 2) I get:
>>>>
>>> It should fix the error.
>>>
>>>>
>>>> 2018-06-18 16:41:40,895: SQL pgadmin: Execute (void) for server #1 -
>>>> DB:ascii (Query-id: 3093186):
>>>> UPDATE public.ascii SET
>>>> data = %(data)s::text WHERE
>>>> id = '2';
>>>> 2018-06-18 16:41:41,027: INFO werkzeug: 127.0.0.1 - - [18/Jun/2018
>>>> 16:41:41] "POST /sqleditor/save/2805058 HTTP/1.1" 500 -
>>>> 2018-06-18 16:41:41,042: ERROR werkzeug: Error on request:
>>>> Traceback (most recent call last):
>>>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/werkzeug/serving.py",
>>>> line 270, in run_wsgi
>>>> execute(self.server.app)
>>>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/werkzeug/serving.py",
>>>> line 258, in execute
>>>> application_iter = app(environ, start_response)
>>>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
>>>> line 1997, in __call__
>>>> return self.wsgi_app(environ, start_response)
>>>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
>>>> line 1985, in wsgi_app
>>>> response = self.handle_exception(e)
>>>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
>>>> line 1540, in handle_exception
>>>> reraise(exc_type, exc_value, tb)
>>>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
>>>> line 1982, in wsgi_app
>>>> response = self.full_dispatch_request()
>>>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
>>>> line 1614, in full_dispatch_request
>>>> rv = self.handle_user_exception(e)
>>>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
>>>> line 1517, in handle_user_exception
>>>> reraise(exc_type, exc_value, tb)
>>>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
>>>> line 1612, in full_dispatch_request
>>>> rv = self.dispatch_request()
>>>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
>>>> line 1598, in dispatch_request
>>>> return self.view_functions[rule.endpoint](**req.view_args)
>>>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask_login.py",
>>>> line 792, in decorated_view
>>>> return func(*args, **kwargs)
>>>> File "/Users/dpage/git/pgadmin4/web/pgadmin/tools/sqleditor/__init__.py",
>>>> line 776, in save
>>>> default_conn)
>>>> File "/Users/dpage/git/pgadmin4/web/pgadmin/tools/sqleditor/command.py",
>>>> line 829, in save
>>>> item['sql'], item['data'])
>>>> File "/Users/dpage/git/pgadmin4/web/pgadmin/utils/driver/psycopg2/connection.py",
>>>> line 975, in execute_void
>>>> self.__internal_blocking_execute(cur, query, params)
>>>> File "/Users/dpage/git/pgadmin4/web/pgadmin/utils/driver/psycopg2/connection.py",
>>>> line 629, in __internal_blocking_execute
>>>> cur.execute(query, params)
>>>> File "/Users/dpage/git/pgadmin4/web/pgadmin/utils/driver/psycopg2/cursor.py",
>>>> line 176, in execute
>>>> return _cursor.execute(self, query, params)
>>>> UnicodeEncodeError: 'ascii' codec can't encode characters in position
>>>> 19-21: ordinal not in range(128)
>>>>
>>>>
>>>>>
>>>>> On Fri, Jun 15, 2018 at 2:33 PM, Aditya Toshniwal <
>>>>> [email protected]> wrote:
>>>>>
>>>>>> Hi Dave,
>>>>>>
>>>>>> On Fri, Jun 15, 2018 at 2:08 PM, Dave Page <[email protected]> wrote:
>>>>>>
>>>>>>> Hi
>>>>>>>
>>>>>>> On Thu, Jun 14, 2018 at 7:05 AM, Aditya Toshniwal <
>>>>>>> [email protected]> wrote:
>>>>>>>
>>>>>>>> I am sorry I missed the attachment. :(
>>>>>>>> PFA.
>>>>>>>>
>>>>>>>
>>>>>>> It looks like the encoding names are getting munged somewhere. I see
>>>>>>> you've accounted for that to some degree in connection.py (you have both
>>>>>>> SQL_ASCII/MULE_INTERNAL and SQLASCII/MULEINTERNAL), however it doesn't seem
>>>>>>> to be enough as I'm getting the following error when trying to download CSV
>>>>>>> from the query tool. Can we ensure that conn.encoding contains an un-munged
>>>>>>> value at all times, or is that coming from psycopg2?
>>>>>>>
>>>>>> That is done by pyscopg2 and conn.encoding is a psycopg2 connection
>>>>>> property.
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> 2018-06-15 09:32:28,799: INFO werkzeug: 127.0.0.1 - - [15/Jun/2018
>>>>>>> 09:32:28] "GET /sqleditor/query_tool/download
>>>>>>> /2732923?query=SELECT%20*%20FROM%20public.sql_ascii%0AORDER%
>>>>>>> 20BY%20id%20ASC%20&filename=sql_ascii.csv HTTP/1.1" 500 -
>>>>>>> 2018-06-15 09:32:28,801: ERROR werkzeug: Error on request:
>>>>>>> Traceback (most recent call last):
>>>>>>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/werkzeug/serving.py",
>>>>>>> line 270, in run_wsgi
>>>>>>> execute(self.server.app)
>>>>>>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/werkzeug/serving.py",
>>>>>>> line 260, in execute
>>>>>>> for data in application_iter:
>>>>>>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/werkzeug/wsgi.py",
>>>>>>> line 870, in __next__
>>>>>>> return self._next()
>>>>>>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/werkzeug/wrappers.py",
>>>>>>> line 82, in _iter_encoded
>>>>>>> for item in iterable:
>>>>>>> File "/Users/dpage/git/pgadmin4/web/pgadmin/utils/driver/psycopg2/connection.py",
>>>>>>> line 752, in gen
>>>>>>> column_name = column_name.decode(conn_encoding)
>>>>>>> LookupError: unknown encoding: SQLASCII
>>>>>>>
>>>>>>
>>>>>> This is because there is code bug here. Below is code used to decode
>>>>>> a column name. Connection encoding and python encoding are two different
>>>>>> things. Python does not know what SQLASCII is. This will work with UTF-8
>>>>>> because python has decoder with same name. I tried to download CSV with the
>>>>>> original code without changes and it fails there too. I will fix this and
>>>>>> will send the updated patch. I should have checked this.
>>>>>> conn_encoding = cur.connection.encoding
>>>>>> column_name = column_name.decode(conn_encoding)
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Dave Page
>>>>>>> Blog: http://pgsnake.blogspot.com
>>>>>>> Twitter: @pgsnake
>>>>>>>
>>>>>>> EnterpriseDB UK: http://www.enterprisedb.com
>>>>>>> The Enterprise PostgreSQL Company
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Thanks and Regards,
>>>>>> Aditya Toshniwal
>>>>>> Software Engineer | EnterpriseDB Software Solutions | Pune
>>>>>> "Don't Complain about Heat, Plant a tree"
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Thanks and Regards,
>>>>> Aditya Toshniwal
>>>>> Software Engineer | EnterpriseDB Software Solutions | Pune
>>>>> "Don't Complain about Heat, Plant a tree"
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Dave Page
>>>> Blog: http://pgsnake.blogspot.com
>>>> Twitter: @pgsnake
>>>>
>>>> EnterpriseDB UK: http://www.enterprisedb.com
>>>> The Enterprise PostgreSQL Company
>>>>
>>>
>>>
>>>
>>> --
>>> Thanks and Regards,
>>> Aditya Toshniwal
>>> Software Engineer | EnterpriseDB Software Solutions | Pune
>>> "Don't Complain about Heat, Plant a tree"
>>>
>>
>>
>>
>> --
>> Dave Page
>> Blog: http://pgsnake.blogspot.com
>> Twitter: @pgsnake
>>
>> EnterpriseDB UK: http://www.enterprisedb.com
>> The Enterprise PostgreSQL Company
>>
>
>
>
> --
> Thanks and Regards,
> Aditya Toshniwal
> Software Engineer | EnterpriseDB Software Solutions | Pune
> "Don't Complain about Heat, Plant a tree"
>
--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake
EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 38+ messages in thread
end of thread, other threads:[~2018-06-21 21:07 UTC | newest]
Thread overview: 38+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2018-05-22 06:05 [pgAdmin4][RM#3289] Can't query SQL_ASCII database. Aditya Toshniwal <[email protected]>
2018-05-22 08:07 ` Dave Page <[email protected]>
2018-05-22 08:46 ` Aditya Toshniwal <[email protected]>
2018-05-22 13:45 ` Victoria Henry <[email protected]>
2018-05-23 04:43 ` Aditya Toshniwal <[email protected]>
2018-05-30 07:06 ` Aditya Toshniwal <[email protected]>
2018-05-30 20:36 ` Joao De Almeida Pereira <[email protected]>
2018-05-31 05:20 ` Aditya Toshniwal <[email protected]>
2018-05-31 13:12 ` Dave Page <[email protected]>
2018-06-04 05:23 ` Aditya Toshniwal <[email protected]>
2018-06-04 09:54 ` Aditya Toshniwal <[email protected]>
2018-06-04 15:24 ` Joao De Almeida Pereira <[email protected]>
2018-06-04 16:44 ` Aditya Toshniwal <[email protected]>
2018-06-04 19:38 ` Joao De Almeida Pereira <[email protected]>
2018-06-05 04:45 ` Aditya Toshniwal <[email protected]>
2018-06-05 08:50 ` Aditya Toshniwal <[email protected]>
2018-06-05 11:26 ` Dave Page <[email protected]>
2018-06-05 12:21 ` Aditya Toshniwal <[email protected]>
2018-06-05 12:35 ` Dave Page <[email protected]>
2018-06-05 12:49 ` Aditya Toshniwal <[email protected]>
2018-06-05 12:53 ` Aditya Toshniwal <[email protected]>
2018-06-05 12:55 ` Dave Page <[email protected]>
2018-06-05 13:03 ` Aditya Toshniwal <[email protected]>
2018-06-05 13:12 ` Dave Page <[email protected]>
2018-06-06 13:02 ` Aditya Toshniwal <[email protected]>
2018-06-07 10:37 ` Dave Page <[email protected]>
2018-06-07 11:05 ` Aditya Toshniwal <[email protected]>
2018-06-07 11:11 ` Dave Page <[email protected]>
2018-06-14 06:04 ` Aditya Toshniwal <[email protected]>
2018-06-14 06:05 ` Aditya Toshniwal <[email protected]>
2018-06-15 08:38 ` Dave Page <[email protected]>
2018-06-15 09:03 ` Aditya Toshniwal <[email protected]>
2018-06-18 13:14 ` Aditya Toshniwal <[email protected]>
2018-06-18 20:53 ` Dave Page <[email protected]>
2018-06-20 14:17 ` Aditya Toshniwal <[email protected]>
2018-06-21 12:57 ` Dave Page <[email protected]>
2018-06-21 15:51 ` Aditya Toshniwal <[email protected]>
2018-06-21 21:07 ` Dave Page <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox