Received: from malur.postgresql.org ([2a02:16a8:dc51::56]) by arkaria.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA384:256) (Exim 4.89) (envelope-from ) id 1fTkdx-00042H-8V for pgadmin-hackers@arkaria.postgresql.org; Fri, 15 Jun 2018 09:03:41 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.89) (envelope-from ) id 1fTkdt-0005KC-UH for pgadmin-hackers@arkaria.postgresql.org; Fri, 15 Jun 2018 09:03:37 +0000 Received: from makus.postgresql.org ([2001:4800:1501:1::229]) by malur.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA384:256) (Exim 4.89) (envelope-from ) id 1fTkdt-0005K2-Fk for pgadmin-hackers@lists.postgresql.org; Fri, 15 Jun 2018 09:03:37 +0000 Received: from mail-lf0-x241.google.com ([2a00:1450:4010:c07::241]) by makus.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1fTkdq-0003V0-2a for pgadmin-hackers@postgresql.org; Fri, 15 Jun 2018 09:03:35 +0000 Received: by mail-lf0-x241.google.com with SMTP id v135-v6so13553651lfa.9 for ; Fri, 15 Jun 2018 02:03:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=enterprisedb-com.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=D+ycvQBKZMHD/NdUp/0zntl9XERIzW/iijCQnjJpw9w=; b=DhDtQTdvWLdpRSIavAutzjd9Q40nLEA+vQgeFveU8UykSw8u2kM9uuyk404EIkJyeg BYUSAmN8WC9kYZkD8G5oNYOhbJ3xM/tssXHjrSmRNNxh/04KWz7RDWxpgoGRrpwF2L0c f2rUiwqCKhKyl/vMqQc0r9hzqu4Hv1gTw8K1aAeXeMkvR66zZGhYm3MoWW3yqxdRCG/o 1CjTJwLzzmQi3RKZaZFOMhydr12oHfC8RW2V3UgMPFIiEEEr8M9npHNECyTVxzXMbRBV tkIkCxCOe26f/Ua3DBhB3x3Ih61o60sizYkHmFvapwETkByIOPYvfXabVEO+m/g8gTIF jAOg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=D+ycvQBKZMHD/NdUp/0zntl9XERIzW/iijCQnjJpw9w=; b=IjxHR5K5+G5J2wlN0OA32x6ukf32eyXyHd0mk6fDd3kO3QuK8DtGqlJrQm2xSbG6cA ZLmpM+DaR7lqimot7kR5UqX10KTcAXZKDR4fuC2aEZv2buGnP2wnpXFp+9mqUCKyf1zE 0srsjf/Pk96ePIHBpRBf/yT7GxQpwSaiOTrXGW7XmUzIw6HpQScTttD/XqwQFiMyK5+B Y1Y5W5PFBkN1q4mAR+wgPIgIL3ibbrXT853LxPkcZQ3L8eh7ABkkX2UNSAsTx+0lvay3 fElofjHFAhzp5zoc+S0B6tvh77WdiKFJ9vder2/Zb9KBDXxBQxx6o/jC5bEQ9DgWxUTx rWpQ== X-Gm-Message-State: APt69E1gsDrveEKU0En4S/5IIdVwY5ANAapZRgyXkxcHZQX0hxAyqlTd 8nvYTKXYHFQnT8UStt5V5kdLWPwh+zDnak4L06pbzQ== X-Google-Smtp-Source: ADUXVKL+ze+YhXfXOmGIehWY5E7mveJM3KwFuQQodSS1RRPbTrC+nBg+pnUhg3V7pvIuvW3X4OY/1s2GPouGeR3E99I= X-Received: by 2002:a2e:f11:: with SMTP id 17-v6mr730772ljp.47.1529053411726; Fri, 15 Jun 2018 02:03:31 -0700 (PDT) MIME-Version: 1.0 Received: by 2002:a2e:7a02:0:0:0:0:0 with HTTP; Fri, 15 Jun 2018 02:03:31 -0700 (PDT) In-Reply-To: References: From: Aditya Toshniwal Date: Fri, 15 Jun 2018 14:33:31 +0530 Message-ID: Subject: Re: [pgAdmin4][RM#3289] Can't query SQL_ASCII database. To: Dave Page Cc: Joao De Almeida Pereira , pgadmin-hackers Content-Type: multipart/alternative; boundary="00000000000072ee18056eaa7daf" List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Precedence: bulk --00000000000072ee18056eaa7daf Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi Dave, On Fri, Jun 15, 2018 at 2:08 PM, Dave Page wrote: > Hi > > On Thu, Jun 14, 2018 at 7:05 AM, Aditya Toshniwal enterprisedb.com> wrote: > >> I am sorry I missed the attachment. :( >> PFA. >> > > It looks like the encoding names are getting munged somewhere. I see > you've accounted for that to some degree in connection.py (you have both > SQL_ASCII/MULE_INTERNAL and SQLASCII/MULEINTERNAL), however it doesn't se= em > to be enough as I'm getting the following error when trying to download C= SV > from the query tool. Can we ensure that conn.encoding contains an un-mung= ed > value at all times, or is that coming from psycopg2? > =E2=80=8BThat is done by pyscopg2 and conn.encoding is a psycopg2 connectio= n property.=E2=80=8B > > 2018-06-15 09:32:28,799: INFO werkzeug: 127.0.0.1 - - [15/Jun/2018 > 09:32:28] "GET /sqleditor/query_tool/download/2732923?query=3DSELECT% > 20*%20FROM%20public.sql_ascii%0AORDER%20BY%20id%20ASC%20&filename=3Dsql_a= scii.csv > HTTP/1.1" 500 - > 2018-06-15 09:32:28,801: ERROR werkzeug: Error on request: > Traceback (most recent call last): > File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/we= rkzeug/serving.py", > line 270, in run_wsgi > execute(self.server.app) > File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/we= rkzeug/serving.py", > line 260, in execute > for data in application_iter: > File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/we= rkzeug/wsgi.py", > line 870, in __next__ > return self._next() > File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site- > packages/werkzeug/wrappers.py", line 82, in _iter_encoded > for item in iterable: > File "/Users/dpage/git/pgadmin4/web/pgadmin/utils/driver/psycopg2/conne= ction.py", > line 752, in gen > column_name =3D column_name.decode(conn_encoding) > LookupError: unknown encoding: SQLASCII > =E2=80=8BThis is because there is code bug here. Below is code used to deco= de a column name. Connection encoding and python encoding are two different things. Python does not know what SQLASCII is. This will work with UTF-8 because python has decoder with same name. I tried to download CSV with the original code without changes and it fails there too. I will fix this and will send the updated patch. I should have checked this. conn_encoding =3D cur.connection.encoding column_name =3D column_name.decode(conn_encoding)=E2=80=8B > > -- > Dave Page > Blog: http://pgsnake.blogspot.com > Twitter: @pgsnake > > EnterpriseDB UK: http://www.enterprisedb.com > The Enterprise PostgreSQL Company > --=20 Thanks and Regards, Aditya Toshniwal Software Engineer | EnterpriseDB Software Solutions | Pune "Don't Complain about Heat, Plant a tree" --00000000000072ee18056eaa7daf Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi Dave,

On Fri, Jun 15, 2018 at 2:08 PM, Dave Page <<= a href=3D"mailto:dpage@pgadmin.org" target=3D"_blank">dpage@pgadmin.org= > wrote:
Hi

On Thu, Jun 14, 2018 at 7:05 AM, Aditya Toshniwa= l <aditya.toshniwal@enterprisedb.com> w= rote:
I am sorry I missed the attachment. :(
PFA.

It looks like the encoding names are getting munge= d somewhere. I see you've accounted for that to some degree in connecti= on.py (you have both SQL_ASCII/MULE_INTERNAL and SQLASCII/MULEINTERNAL), ho= wever it doesn't seem to be enough as I'm getting the following err= or when trying to download CSV from the query tool. Can we ensure that conn= .encoding contains an un-munged value at all times, or is that coming from = psycopg2?
=E2=80= =8BThat is done by pyscopg2 and conn.encoding is a psycopg2 connection prop= erty.=E2=80=8B
=C2=A0

2018-06-15 09:32:28,799: INFO werkzeug: 127.0.0.1 - - [15/Jun/2018 09:32:28] "GET /sqleditor/quer= y_tool/download/2732923?query=3DSELECT%20*%20FROM%20public.sql_as= cii%0AORDER%20BY%20id%20ASC%20&filename=3Dsql_ascii.csv HTTP/= 1.1" 500 -
2018-06-15 09:32:28,801: ERROR werkzeug: Error on request:
Traceback (most rec= ent call last):
=C2=A0 File "/Users/dpage/.virtualenv= s/pgadmin4/lib/python2.7/site-packages/werkzeug/serving.py",= line 270, in run_wsgi
=C2=A0 =C2=A0 execute(self.server.app)
=C2=A0 File "/Users/dpage/.virtualenvs/pgadmin4/lib/python= 2.7/site-packages/werkzeug/serving.py", line 260, in execute
=C2=A0 =C2=A0 for data in application_iter:
=C2=A0 File &qu= ot;/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages= /werkzeug/wsgi.py", line 870, in __next__
=C2=A0 =C2=A0 retu= rn self._next()
=C2=A0 File "/Users/dpage/.virtualenvs/= pgadmin4/lib/python2.7/site-packages/werkzeug/wrappers.py", = line 82, in _iter_encoded
=C2=A0 =C2=A0 for item in iterable:
=C2=A0 File "/Users/dpage/git/pgadmin4/web/pgadmin/utils/d= river/psycopg2/connection.py", line 752, in gen
=C2=A0 = =C2=A0 column_name =3D column_name.decode(conn_encoding)
Loo= kupError: unknown encoding: SQLASCII
=C2=A0
=E2=80=8BThis is because there is= code bug here. Below is code used to decode a column name. Connection enco= ding and python encoding are two different things. Python does not know wha= t SQLASCII is. This will work with UTF-8 because python has decoder with sa= me name. I tried to download CSV with the original code without changes and= it fails there too. I will fix this and will send the updated patch. I sho= uld have checked this.
conn_encoding =3D cur.co= nnection.encoding
column_name =3D column_name.dec= ode(conn_encoding)=E2=80=8B
=C2=A0

--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnak= e

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Compa= ny



--
Thanks and Regards,
Aditya Toshniwal
Software En= gineer |=C2=A0EnterpriseDB Software Solutions |=C2=A0Pune
"Do= n't Complain about Heat, Plant a tree"
--00000000000072ee18056eaa7daf--