public inbox for [email protected]
help / color / mirror / Atom feedFrom: Aditya Toshniwal <[email protected]>
To: Dave Page <[email protected]>
Cc: Joao De Almeida Pereira <[email protected]>
Cc: pgadmin-hackers <[email protected]>
Subject: Re: [pgAdmin4][RM#3289] Can't query SQL_ASCII database.
Date: Thu, 21 Jun 2018 21:21:43 +0530
Message-ID: <CAM9w-_=6V+6q3JUktmfR_7fw3uUZvj4fMme=uEH_38S3uKr5=g@mail.gmail.com> (raw)
In-Reply-To: <CA+OCxoz3+7PuS4pfKUfoGHPkwLDGGhaLWdgO_xZzUD1jii0xbg@mail.gmail.com>
References: <CAM9w-_ntCUguZp2rVJXPsDvWJbWneAz4-AHJoYB+nUTSYMXc9g@mail.gmail.com>
<CA+OCxozRjJY6RWrXpMLD6-dWnm2LdPs6bS+VmJa403vxVoLwhg@mail.gmail.com>
<CAM9w-_m-cJax16zqBonx-d0-HSy5dNoOzB13HWAad5Xv0Wu=CA@mail.gmail.com>
<CANxYE3+qrGeZYzdYEtzxS8i2P+325V5E+n1xVpyzTwn7vRqRDg@mail.gmail.com>
<CAM9w-_mTe0gW+2cdhMutMWyBsRt7Om3G_Em126Ryntf0nOiHbA@mail.gmail.com>
<CAM9w-_mXdJhuRFn87yAuqWtfrhnRgw1KvU2PpthL-hbnOt2QpA@mail.gmail.com>
<CAE+jjampAPoDq7xJgeNf1L=i_1jf83PLr=dZYicfkEMp6_UBAg@mail.gmail.com>
<CAM9w-_k2Fo-qDy+qxM5i=Aba__TAWxrryR+m3izTXB=koToKcQ@mail.gmail.com>
<CA+OCxoz+Wo3pUuu8SMBzAsQvrOjYLPQHCfQMtO3X3HKezKcapw@mail.gmail.com>
<CAM9w-_nEDycbeEEeUhKwXjxmkNC39YzqhggJR3Frc4pq6WcJ0Q@mail.gmail.com>
<CAM9w-_kDQkAWRZrRRtmMDz8DZ5x2wGoUro6oQbQqTzLmZZGyBw@mail.gmail.com>
<CAE+jjametYGjStNFZFW544Jcm_by1OABtb7yFQWKGzyfk08QiA@mail.gmail.com>
<CAM9w-_=rsazJWCu5xRqLGkYX3FW22vVuyc0NcR5TPskO9PApUg@mail.gmail.com>
<CAE+jjakwbfsbwkvwa6N1QGjGeVb36sWnde+C6amnHQLHdTE3kQ@mail.gmail.com>
<CAM9w-_km5Zv=Gw6CUZF8m3zzacOKL-ir-QNw3xPjP2MJfLRg7w@mail.gmail.com>
<CAM9w-_=c=82+5ESAEMXUQf5NOrfouZ5JxJMfr+56vJAfGz2DrA@mail.gmail.com>
<CA+OCxoxAhWEF5tQQy84QoOYB=ed--o--ppBoXZ1dwvWNBoNMQw@mail.gmail.com>
<CAM9w-_nUfackUWYNZDRcJ43D6ftYWnWkRsxvcyZVDzABkqza8A@mail.gmail.com>
<CA+OCxoyRgrQ=83Wo1YwsR9oSHE87=-20uP7d7zmF9F685CW2sA@mail.gmail.com>
<CAM9w-_ngHm2XpfXeA+RtWhifsHmN06rOEYwg5UrZBc=ZOum1Ug@mail.gmail.com>
<CA+OCxoyn32gZu78cc-p7Cr3zdsyviBVYD+X59W0-u+DTVUOzJg@mail.gmail.com>
<CAM9w-_=xKk35uQBYM+yZGkttryodU1CPF4n=jYEezKDm4GD_qw@mail.gmail.com>
<CA+OCxoxuS2GXK=LxREpniK6hHQ=d2G2FBpjD7JSohxZJpQwxPA@mail.gmail.com>
<CAM9w-_kPsWEodXtznzA0ZwXSO3d8t2agg5sUzF=2EEsGfakuMw@mail.gmail.com>
<CA+OCxoz7fFQn5ADzLbohXknMNk-1kfeTjAgCO4NrsXZPRiFVNw@mail.gmail.com>
<CAM9w-_mk04Rz6KVvQtmwiHs4s+OFDsDgCKebUO-vT0W2DfGi4w@mail.gmail.com>
<CA+OCxoyQtqqWhfNY5Um907AocAt1GsLcWvFJGjr5FLLeoWDCkA@mail.gmail.com>
<CAM9w-_n6LtJTXYhmrcYqvyvzXkshDMM3Z0yyKUWU4Vdmg7tNvw@mail.gmail.com>
<CAM9w-_=g0GJTyF8QegHJtKojcghcE+xzDPQUig_YzUnnH3nyWg@mail.gmail.com>
<CA+OCxox-DmkBptAXwiwi_GMCsow0dXzgE2C-wOyMZqGmUd3vCQ@mail.gmail.com>
<CAM9w-_niwZ+VD4awqOLp1D5eEFtE_G8wWciWC_sg8N2yvER=Nw@mail.gmail.com>
<CAM9w-_=8k4gBg_kji7KQ7FO=j18tCU5kDzu1YGGF3Xw7XEfK7A@mail.gmail.com>
<CA+OCxowxJghshFjXCtOUiRa0qqS93pC4m-bPRhdb7Q3AWe+HnQ@mail.gmail.com>
<CAM9w-_k7FeHMwcOd1RqkCC64pF2b3=WHXpM14YJZ0b45tWUdBw@mail.gmail.com>
<CA+OCxoz3+7PuS4pfKUfoGHPkwLDGGhaLWdgO_xZzUD1jii0xbg@mail.gmail.com>
Hi Hackers,
PFA patch to make SQL ASCII related changes compatible with Python 2.6.
Dictionary comprehension is not supported in Python 2.6.
On Thu, Jun 21, 2018 at 6:27 PM, Dave Page <[email protected]> wrote:
> Thanks - patch applied!
>
> On Wed, Jun 20, 2018 at 3:17 PM, Aditya Toshniwal <aditya.toshniwal@
> enterprisedb.com> wrote:
>
>> Hi Dave,
>>
>> Attached is the updated patch. (Playing with encodings is not at all fun
>> :( )
>>
>> On Tue, Jun 19, 2018 at 2:23 AM, Dave Page <[email protected]> wrote:
>>
>>> Hi
>>>
>>> On Mon, Jun 18, 2018 at 2:14 PM, Aditya Toshniwal <
>>> [email protected]> wrote:
>>>
>>>> Hi Hackers,
>>>>
>>>> Attached is the updated patch which includes the fix for Download CSV
>>>> fail in SQL_ASCII database, which is RM3250
>>>> <https://redmine.postgresql.org/issues/3250;
>>>> This should fix RM3289 and RM3250. As they interrelated, sending the
>>>> combined patch.
>>>> Kindly review.
>>>>
>>>
>>> This is definitely looking better - both view and save now work as
>>> expected. However, using the test data the I posted upthread, if I try to
>>> edit a value (in this case by adding a couple of chars to the end of the
>>> data in row 2) I get:
>>>
>> It should fix the error.
>>
>>>
>>> 2018-06-18 16:41:40,895: SQL pgadmin: Execute (void) for server #1 -
>>> DB:ascii (Query-id: 3093186):
>>> UPDATE public.ascii SET
>>> data = %(data)s::text WHERE
>>> id = '2';
>>> 2018-06-18 16:41:41,027: INFO werkzeug: 127.0.0.1 - - [18/Jun/2018
>>> 16:41:41] "POST /sqleditor/save/2805058 HTTP/1.1" 500 -
>>> 2018-06-18 16:41:41,042: ERROR werkzeug: Error on request:
>>> Traceback (most recent call last):
>>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/werkzeug/serving.py",
>>> line 270, in run_wsgi
>>> execute(self.server.app)
>>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/werkzeug/serving.py",
>>> line 258, in execute
>>> application_iter = app(environ, start_response)
>>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
>>> line 1997, in __call__
>>> return self.wsgi_app(environ, start_response)
>>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
>>> line 1985, in wsgi_app
>>> response = self.handle_exception(e)
>>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
>>> line 1540, in handle_exception
>>> reraise(exc_type, exc_value, tb)
>>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
>>> line 1982, in wsgi_app
>>> response = self.full_dispatch_request()
>>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
>>> line 1614, in full_dispatch_request
>>> rv = self.handle_user_exception(e)
>>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
>>> line 1517, in handle_user_exception
>>> reraise(exc_type, exc_value, tb)
>>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
>>> line 1612, in full_dispatch_request
>>> rv = self.dispatch_request()
>>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
>>> line 1598, in dispatch_request
>>> return self.view_functions[rule.endpoint](**req.view_args)
>>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask_login.py",
>>> line 792, in decorated_view
>>> return func(*args, **kwargs)
>>> File "/Users/dpage/git/pgadmin4/web/pgadmin/tools/sqleditor/__init__.py",
>>> line 776, in save
>>> default_conn)
>>> File "/Users/dpage/git/pgadmin4/web/pgadmin/tools/sqleditor/command.py",
>>> line 829, in save
>>> item['sql'], item['data'])
>>> File "/Users/dpage/git/pgadmin4/web/pgadmin/utils/driver/psycopg2/connection.py",
>>> line 975, in execute_void
>>> self.__internal_blocking_execute(cur, query, params)
>>> File "/Users/dpage/git/pgadmin4/web/pgadmin/utils/driver/psycopg2/connection.py",
>>> line 629, in __internal_blocking_execute
>>> cur.execute(query, params)
>>> File "/Users/dpage/git/pgadmin4/web/pgadmin/utils/driver/psycopg2/cursor.py",
>>> line 176, in execute
>>> return _cursor.execute(self, query, params)
>>> UnicodeEncodeError: 'ascii' codec can't encode characters in position
>>> 19-21: ordinal not in range(128)
>>>
>>>
>>>>
>>>> On Fri, Jun 15, 2018 at 2:33 PM, Aditya Toshniwal <
>>>> [email protected]> wrote:
>>>>
>>>>> Hi Dave,
>>>>>
>>>>> On Fri, Jun 15, 2018 at 2:08 PM, Dave Page <[email protected]> wrote:
>>>>>
>>>>>> Hi
>>>>>>
>>>>>> On Thu, Jun 14, 2018 at 7:05 AM, Aditya Toshniwal <
>>>>>> [email protected]> wrote:
>>>>>>
>>>>>>> I am sorry I missed the attachment. :(
>>>>>>> PFA.
>>>>>>>
>>>>>>
>>>>>> It looks like the encoding names are getting munged somewhere. I see
>>>>>> you've accounted for that to some degree in connection.py (you have both
>>>>>> SQL_ASCII/MULE_INTERNAL and SQLASCII/MULEINTERNAL), however it doesn't seem
>>>>>> to be enough as I'm getting the following error when trying to download CSV
>>>>>> from the query tool. Can we ensure that conn.encoding contains an un-munged
>>>>>> value at all times, or is that coming from psycopg2?
>>>>>>
>>>>> That is done by pyscopg2 and conn.encoding is a psycopg2 connection
>>>>> property.
>>>>>
>>>>>
>>>>>>
>>>>>> 2018-06-15 09:32:28,799: INFO werkzeug: 127.0.0.1 - - [15/Jun/2018
>>>>>> 09:32:28] "GET /sqleditor/query_tool/download
>>>>>> /2732923?query=SELECT%20*%20FROM%20public.sql_ascii%0AORDER%
>>>>>> 20BY%20id%20ASC%20&filename=sql_ascii.csv HTTP/1.1" 500 -
>>>>>> 2018-06-15 09:32:28,801: ERROR werkzeug: Error on request:
>>>>>> Traceback (most recent call last):
>>>>>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/werkzeug/serving.py",
>>>>>> line 270, in run_wsgi
>>>>>> execute(self.server.app)
>>>>>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/werkzeug/serving.py",
>>>>>> line 260, in execute
>>>>>> for data in application_iter:
>>>>>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/werkzeug/wsgi.py",
>>>>>> line 870, in __next__
>>>>>> return self._next()
>>>>>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/werkzeug/wrappers.py",
>>>>>> line 82, in _iter_encoded
>>>>>> for item in iterable:
>>>>>> File "/Users/dpage/git/pgadmin4/web/pgadmin/utils/driver/psycopg2/connection.py",
>>>>>> line 752, in gen
>>>>>> column_name = column_name.decode(conn_encoding)
>>>>>> LookupError: unknown encoding: SQLASCII
>>>>>>
>>>>>
>>>>> This is because there is code bug here. Below is code used to decode
>>>>> a column name. Connection encoding and python encoding are two different
>>>>> things. Python does not know what SQLASCII is. This will work with UTF-8
>>>>> because python has decoder with same name. I tried to download CSV with the
>>>>> original code without changes and it fails there too. I will fix this and
>>>>> will send the updated patch. I should have checked this.
>>>>> conn_encoding = cur.connection.encoding
>>>>> column_name = column_name.decode(conn_encoding)
>>>>>
>>>>>
>>>>>>
>>>>>> --
>>>>>> Dave Page
>>>>>> Blog: http://pgsnake.blogspot.com
>>>>>> Twitter: @pgsnake
>>>>>>
>>>>>> EnterpriseDB UK: http://www.enterprisedb.com
>>>>>> The Enterprise PostgreSQL Company
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Thanks and Regards,
>>>>> Aditya Toshniwal
>>>>> Software Engineer | EnterpriseDB Software Solutions | Pune
>>>>> "Don't Complain about Heat, Plant a tree"
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Thanks and Regards,
>>>> Aditya Toshniwal
>>>> Software Engineer | EnterpriseDB Software Solutions | Pune
>>>> "Don't Complain about Heat, Plant a tree"
>>>>
>>>
>>>
>>>
>>> --
>>> Dave Page
>>> Blog: http://pgsnake.blogspot.com
>>> Twitter: @pgsnake
>>>
>>> EnterpriseDB UK: http://www.enterprisedb.com
>>> The Enterprise PostgreSQL Company
>>>
>>
>>
>>
>> --
>> Thanks and Regards,
>> Aditya Toshniwal
>> Software Engineer | EnterpriseDB Software Solutions | Pune
>> "Don't Complain about Heat, Plant a tree"
>>
>
>
>
> --
> Dave Page
> Blog: http://pgsnake.blogspot.com
> Twitter: @pgsnake
>
> EnterpriseDB UK: http://www.enterprisedb.com
> The Enterprise PostgreSQL Company
>
--
Thanks and Regards,
Aditya Toshniwal
Software Engineer | EnterpriseDB Software Solutions | Pune
"Don't Complain about Heat, Plant a tree"
Attachments:
[application/octet-stream] RM3289_py26.patch (1.4K, 3-RM3289_py26.patch)
download | inline diff:
diff --git a/web/pgadmin/utils/driver/psycopg2/connection.py b/web/pgadmin/utils/driver/psycopg2/connection.py
index e8ed88a3..1b4d2a3f 100644
--- a/web/pgadmin/utils/driver/psycopg2/connection.py
+++ b/web/pgadmin/utils/driver/psycopg2/connection.py
@@ -622,14 +622,15 @@ WHERE
# We need to esacpe the data so that it does not fail when
# it is encoded with python ascii
# unicode_escape helps in escaping and unescaping
- if self.conn.encoding in ('SQL_ASCII', 'SQLASCII',
- 'MULE_INTERNAL', 'MULEINTERNAL')\
- and params is not None and type(params) == dict:
- params = {
- key: val.encode('unicode_escape')
- .decode('raw_unicode_escape')
- for key, val in params.items()
- }
+ if self.conn:
+ if self.conn.encoding in ('SQL_ASCII', 'SQLASCII',
+ 'MULE_INTERNAL', 'MULEINTERNAL')\
+ and params is not None and type(params) == dict:
+ params = dict(
+ (key, val.encode('unicode_escape')
+ .decode('raw_unicode_escape'))
+ for key, val in params.items()
+ )
return params
def __internal_blocking_execute(self, cur, query, params):
view thread (38+ messages) latest in thread
reply
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: [email protected]
Cc: [email protected], [email protected], [email protected]
Subject: Re: [pgAdmin4][RM#3289] Can't query SQL_ASCII database.
In-Reply-To: <CAM9w-_=6V+6q3JUktmfR_7fw3uUZvj4fMme=uEH_38S3uKr5=g@mail.gmail.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox