Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtp (Exim 4.84_2) (envelope-from ) id 1bNAUT-0005oO-EC for pgadmin-hackers@arkaria.postgresql.org; Wed, 13 Jul 2016 03:05:37 +0000 Received: from localhost ([127.0.0.1] helo=postgresql.org) by malur.postgresql.org with smtp (Exim 4.84_2) (envelope-from ) id 1bNAUS-0001Mp-Ft for pgadmin-hackers@arkaria.postgresql.org; Wed, 13 Jul 2016 03:05:36 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1bNAUF-000192-0J for pgadmin-hackers@postgresql.org; Wed, 13 Jul 2016 03:05:23 +0000 Received: from mail-io0-x236.google.com ([2607:f8b0:4001:c06::236]) by magus.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.84_2) (envelope-from ) id 1bNAU4-0003CO-6C for pgadmin-hackers@postgresql.org; Wed, 13 Jul 2016 03:05:20 +0000 Received: by mail-io0-x236.google.com with SMTP id b62so34687732iod.3 for ; Tue, 12 Jul 2016 20:05:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=enterprisedb-com.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=rQ1QQzeO0zDX3CUdBxuSTiKmNEUxhpLvaSR5BWI4j7k=; b=eRONAiLk7ZxagddOEFfyocWORAnNxI0Rf0MgunWMV0xytXymweOmpgI1HRJob5HyWg 0GKgyj4w07erTWqEyTEpkGDSuWBotyTArFOqdkfrBYqPrRtcJsV1gGy/ffCdyg3hQ2OD R7dlNWz5CUUHGzUyyLNFosjRKrcQ/zhw++f6pU/uk+cYMzMwPIl6vfhNE0GR4dDF0K01 8eGzAdBccgpl9/8FvUXcwpFB7uP6Y9mkbO0sKS+j0gyaOylpYHMlU68HDXKYHF/StlZz BIHuQxrpvjL2zcdaKQ0eq4ROR16aS1tG9AaJ8fsXnGpJNTd1GpXwWD0P+4ZaYYkB1nQJ vhmw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=rQ1QQzeO0zDX3CUdBxuSTiKmNEUxhpLvaSR5BWI4j7k=; b=L2ssCzzo5NjpQEF1yEe9xISzsBbI9griNg+DAXaAnYuaEvVNmtLSo0j+c/hyZ1OQhK rwg58bMDsVZ/m646t/G8pFnojRARfZTD5sgEVeOZOR6KS/KReb6TSz2oIKZY4vVnKq5h LKupgm2XVkQmtyKimOf53vH6bKF7IbsShhhJUOJjyqN0urWANUf9zOK7ZfiykiOztneB 1S+5p/lnMFNEXRFMbbUdpRe8VUAkQKu9TYhYCw/BkEetcXSArH6+YFgz4nlITcvSTalk v8U5Kzni8RhKENcDs8wGq7FGp/lsPRbBbNipAiQoTo5PKcVieZSS4k7dxNmFt826Q82u 7nGA== X-Gm-Message-State: ALyK8tLndtv5xYf6UX5gtfMhPcxVeyroyTR1o4cgnusAf3/PlqBWv173VeTzly0MGVzc7iNxj6H9Y4GdxpTMRhre X-Received: by 10.107.41.75 with SMTP id p72mr7187901iop.106.1468379109646; Tue, 12 Jul 2016 20:05:09 -0700 (PDT) MIME-Version: 1.0 Received: by 10.107.6.76 with HTTP; Tue, 12 Jul 2016 20:04:50 -0700 (PDT) In-Reply-To: <5aef049f-d684-3a52-5b8f-0b31e3a74f32@gmail.com> References: <2048fa7e-7563-3edd-db2c-d58b97b98bee@gmail.com> <5aef049f-d684-3a52-5b8f-0b31e3a74f32@gmail.com> From: Ashesh Vashi Date: Wed, 13 Jul 2016 08:34:50 +0530 Message-ID: Subject: Re: [Patch] Fix Unicode in errmsg To: Magnun Leno Cc: pgadmin-hackers Content-Type: multipart/alternative; boundary=001a1141ef5239fcf505377ba7a8 X-Pg-Spam-Score: -2.6 (--) List-Archive: List-Help: List-ID: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: X-Mailing-List: pgadmin-hackers Precedence: bulk Sender: pgadmin-hackers-owner@postgresql.org --001a1141ef5239fcf505377ba7a8 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable On Wed, Jul 13, 2016 at 1:11 AM, Magnun Leno wrote: > What error are you getting? > Can you please share the exact error message to understand the issue > properly? > > Sorry, my bad. Here is the traceback: > > 2016-07-12 16:12:22,366: ERROR pgadmin: Exception on > /browser/server-group/obj/ [POST] > Traceback (most recent call last): > File > "/home/magnun/venv/pgadmin4/lib/python2.7/site-packages/flask/app.py", li= ne > 1817, in wsgi_app > response =3D self.full_dispatch_request() > File > "/home/magnun/venv/pgadmin4/lib/python2.7/site-packages/flask/app.py", li= ne > 1477, in full_dispatch_request > rv =3D self.handle_user_exception(e) > File > "/home/magnun/venv/pgadmin4/lib/python2.7/site-packages/flask/app.py", li= ne > 1381, in handle_user_exception > reraise(exc_type, exc_value, tb) > File > "/home/magnun/venv/pgadmin4/lib/python2.7/site-packages/flask/app.py", li= ne > 1475, in full_dispatch_request > rv =3D self.dispatch_request() > File > "/home/magnun/venv/pgadmin4/lib/python2.7/site-packages/flask/app.py", li= ne > 1461, in dispatch_request > return self.view_functions[rule.endpoint](**req.view_args) > File > "/home/magnun/venv/pgadmin4/lib/python2.7/site-packages/flask/views.py", > line 84, in view > return self.dispatch_request(*args, **kwargs) > File > "/home/magnun/venv/pgadmin4/lib/python2.7/site-packages/pgadmin4/pgadmin/= browser/utils.py", > line 235, in dispatch_request > return method(*args, **kwargs) > File > "/home/magnun/venv/pgadmin4/lib/python2.7/site-packages/pgadmin4/pgadmin/= browser/server_groups/__init__.py", > line 204, in create > data =3D request.form if request.form else > json.loads(request.data.decode()) > UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 14: > ordinal not in range(128) > > How to reproduce: > > 1. Start PgAdmin4; > 2. Access localhost:5050 > 3. Login > 4. In the left panel, right click in Servers. Then, "Create > Server > Group", or "Create > Server" (both share the same error) > 5. In the "Name field", write any non ASCII word. In my case I was > using the following name: Produ=C3=A7=C3=A3o > 6. Click "Save" > > More info about my env: > > $ env | grep LANG > LANG=3Dpt_BR.UTF-8 > GDM_LANG=3Dpt_BR.UTF-8 > > $ python --version > Python 2.7.11 > > $ pip freeze > Babel=3D=3D1.3 > beautifulsoup4=3D=3D4.4.1 > blinker=3D=3D1.3 > django-htmlmin=3D=3D0.8.0 > extras=3D=3D0.0.3 > fixtures=3D=3D2.0.0 > Flask=3D=3D0.10.1 > Flask-Babel=3D=3D0.9 > Flask-Gravatar=3D=3D0.4.2 > Flask-Login=3D=3D0.2.11 > Flask-Mail=3D=3D0.9.1 > Flask-Principal=3D=3D0.4.0 > Flask-Security=3D=3D1.7.4 > Flask-SQLAlchemy=3D=3D2.0 > Flask-WTF=3D=3D0.11 > html5lib=3D=3D1.0b3 > importlib=3D=3D1.0.3 > itsdangerous=3D=3D0.24 > Jinja2=3D=3D2.7.3 > linecache2=3D=3D1.0.0 > MarkupSafe=3D=3D0.23 > passlib=3D=3D1.6.2 > pbr=3D=3D1.9.1 > pgadmin4=3D=3D1.0b2 > psycopg2=3D=3D2.5.2 > pycrypto=3D=3D2.6.1 > pyrsistent=3D=3D0.11.13 > python-dateutil=3D=3D2.5.0 > python-mimeparse=3D=3D1.5.1 > pytz=3D=3D2014.10 > simplejson=3D=3D3.6.5 > six=3D=3D1.9.0 > speaklater=3D=3D1.3 > SQLAlchemy=3D=3D0.9.8 > sqlparse=3D=3D0.1.19 > testscenarios=3D=3D0.5.0 > testtools=3D=3D2.0.0 > traceback2=3D=3D1.4.0 > unittest2=3D=3D1.1.0 > Werkzeug=3D=3D0.9.6 > WTForms=3D=3D2.0.2 > > > We will need to change too many places by this logic. > I've attached patch for it. > > I'm not familiar with the code base yet, but not necessarily all > occurrences of "decode()" need to be patched, just the ones that deal wit= h > user input. > I've changed only the data coming as the user input for deferent nodes. :-) > > I've done some further testing and noticed that even running an insert or > update with non ASCII text runs into failure. > I've not yet looked at the other utilities in pgAdmin IV. I'll look in to it too, and share the updated patch. > This "update users set cidade=3D'Bras=C3=ADlia' where id=3D1;", results i= n the > following traceback: > > 2016-07-12 16:32:59,964: ERROR pgadmin: Exception on > /sqleditor/query_tool/start/2760583 [POST] > Traceback (most recent call last): > File > "/home/magnun/venv/pgadmin4/lib/python2.7/site-packages/flask/app.py", li= ne > 1817, in wsgi_app > response =3D self.full_dispatch_request() > File > "/home/magnun/venv/pgadmin4/lib/python2.7/site-packages/flask/app.py", li= ne > 1477, in full_dispatch_request > rv =3D self.handle_user_exception(e) > File > "/home/magnun/venv/pgadmin4/lib/python2.7/site-packages/flask/app.py", li= ne > 1381, in handle_user_exception > reraise(exc_type, exc_value, tb) > File > "/home/magnun/venv/pgadmin4/lib/python2.7/site-packages/flask/app.py", li= ne > 1475, in full_dispatch_request > rv =3D self.dispatch_request() > File > "/home/magnun/venv/pgadmin4/lib/python2.7/site-packages/flask/app.py", li= ne > 1461, in dispatch_request > return self.view_functions[rule.endpoint](**req.view_args) > File > "/home/magnun/venv/pgadmin4/lib/python2.7/site-packages/flask_login.py", > line 758, in decorated_view > return func(*args, **kwargs) > File > "/home/magnun/venv/pgadmin4/lib/python2.7/site-packages/pgadmin4/pgadmin/= tools/sqleditor/__init__.py", > line 241, in start_query_tool > sql =3D json.loads(request.data.decode()) > UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 48: > ordinal not in range(128) > > But, the other way around isn't true. Executing the update mentioned abov= e > in psql and then running a select in PgAdmin4 (select * from users where > id=3D1) works fine. > > I'll give it a try with Python 3 as soon as possible, it might solve all > the issues since it uses unicode by default. > It's good to have feedback like above. That will give us the idea, where we need improvements, as all the current developers, working on pgAdmin 4, are from English background. Thanks for the inputs/feedbacks, please keep them coming. -- Thanks & Regards, Ashesh Vashi EnterpriseDB INDIA: Enterprise PostgreSQL Company *http://www.linkedin.com/in/asheshvashi* > > Thanks in advance. > > --- > Magnun Leno > > --001a1141ef5239fcf505377ba7a8 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable

On Wed, Jul 13, 2016 at 1:11 AM, Ma= gnun Leno <magnun.leno@gmail.com> wrote:

=20 =20 =20
What error are you getting?
Can you please share the exact error message to understand the issue properly?
 Sorry, my bad. Here is the traceback:

2016-07-12 16:12:22,366: ERROR=C2=A0=C2=A0=C2=A0 pgadmin:=C2=A0=C2= =A0=C2=A0 Exception on /browser/server-group/obj/ [POST]
Traceback (most recent call last):
=C2=A0 File "/home/magnun/venv/pgadmin4/lib/python2.7/site-packages/flask/ap= p.py", line 1817, in wsgi_app
=C2=A0=C2=A0=C2=A0 response =3D self.full_dispatch_request()
=C2=A0 File "/home/magnun/venv/pgadmin4/lib/python2.7/site-packages/flask/ap= p.py", line 1477, in full_dispatch_request
=C2=A0=C2=A0=C2=A0 rv =3D self.handle_user_exception(e)
=C2=A0 File "/home/magnun/venv/pgadmin4/lib/python2.7/site-packages/flask/ap= p.py", line 1381, in handle_user_exception
=C2=A0=C2=A0=C2=A0 reraise(exc_type, exc_value, tb) =C2=A0 File "/home/magnun/venv/pgadmin4/lib/python2.7/site-packages/flask/ap= p.py", line 1475, in full_dispatch_request
=C2=A0=C2=A0=C2=A0 rv =3D self.dispatch_request()
=C2=A0 File "/home/magnun/venv/pgadmin4/lib/python2.7/site-packages/flask/ap= p.py", line 1461, in dispatch_request
=C2=A0=C2=A0=C2=A0 return self.view_functions[rule.endpoint](**req.view_args)
=C2=A0 File "/home/magnun/venv/pgadmin4/lib/python2.7/site-packages/flask/vi= ews.py", line 84, in view
=C2=A0=C2=A0=C2=A0 return self.dispatch_request(*args, **kwarg= s)
=C2=A0 File "/home/magnun/venv/pgadmin4/lib/python2.7/site-packages/pgadmin4/pgadm= in/browser/utils.py", line 235, in dispatch_request
=C2=A0=C2=A0=C2=A0 return method(*args, **kwargs)
=C2=A0 File "/home/magnun/venv/pgadmin4/lib/python2.7/site-packages/pgadmin4/pgadm= in/browser/server_groups/__init__.py", line 204, in create
=C2=A0=C2=A0=C2=A0 data =3D request.form if request.form else json.loads(request.data.decode())
UnicodeDecodeError: 'ascii' codec can't decode byt= e 0xc3 in position 14: ordinal not in range(128)

How to reproduce:
  1. Start PgAdmin4;
  2. Access localhost:5050
  3. Login
  4. In the left panel, right click in Servers. Then, "Create >= ; Server Group", or "Create > Server" (both share t= he same error)
  5. In the "Name field", write any non ASCII word. In my ca= se I was using the following name: Produ=C3=A7=C3=A3o
  6. Click "Save"
More info about my env:

$ env | grep LANG
LANG=3Dpt_BR.UTF-8
GDM_LANG=3Dpt_BR.UTF-8

$ python --version
Python 2.7.11

$ pip freeze
Babel=3D=3D1.3
beautifulsoup4=3D=3D4.4.1
blinker=3D=3D1.3
django-htmlmin=3D=3D0.8.0
extras=3D=3D0.0.3
fixtures=3D=3D2.0.0
Flask=3D=3D0.10.1
Flask-Babel=3D=3D0.9
Flask-Gravatar=3D=3D0.4.2
Flask-Login=3D=3D0.2.11
Flask-Mail=3D=3D0.9.1
Flask-Principal=3D=3D0.4.0
Flask-Security=3D=3D1.7.4
Flask-SQLAlchemy=3D=3D2.0
Flask-WTF=3D=3D0.11
html5lib=3D=3D1.0b3
importlib=3D=3D1.0.3
itsdangerous=3D=3D0.24
Jinja2=3D=3D2.7.3
linecache2=3D=3D1.0.0
MarkupSafe=3D=3D0.23
passlib=3D=3D1.6.2
pbr=3D=3D1.9.1
pgadmin4=3D=3D1.0b2
psycopg2=3D=3D2.5.2
pycrypto=3D=3D2.6.1
pyrsistent=3D=3D0.11.13
python-dateutil=3D=3D2.5.0
python-mimeparse=3D=3D1.5.1
pytz=3D=3D2014.10
simplejson=3D=3D3.6.5
six=3D=3D1.9.0
speaklater=3D=3D1.3
SQLAlchemy=3D=3D0.9.8
sqlparse=3D=3D0.1.19
testscenarios=3D=3D0.5.0
testtools=3D=3D2.0.0
traceback2=3D=3D1.4.0
unittest2=3D=3D1.1.0
Werkzeug=3D=3D0.9.6
WTForms=3D=3D2.0.2


We will need to change too many places by this logic.
I've attached patch for it.
 I'm not familiar with the code base yet, but not necessarily all occurrences of "decode()" need to be patched, just the ones t= hat deal with user input.
I've changed only = the data coming as the user input for deferent nodes. :-)

I've done some further testing and noticed that even running an insert or update with non ASCII text runs into failure.
I've not yet looked at the other utilities in pgAdmin IV.
=

I'll look in to it too, and share the updated patch= .=C2=A0
Th= is "update users set cidade=3D'Bras=C3=ADlia' where id=3D1;", = results in the following traceback:

2016-07-12 16:32:59,964: ERROR=C2=A0=C2=A0=C2=A0 pgadmin:=C2=A0=C2= =A0=C2=A0 Exception on /sqleditor/query_tool/start/2760583 [POST]
Traceback (most recent call last):
=C2=A0 File "/home/magnun/venv/pgadmin4/lib/python2.7/site-packages/flask/ap= p.py", line 1817, in wsgi_app
=C2=A0=C2=A0=C2=A0 response =3D self.full_dispatch_request()
=C2=A0 File "/home/magnun/venv/pgadmin4/lib/python2.7/site-packages/flask/ap= p.py", line 1477, in full_dispatch_request
=C2=A0=C2=A0=C2=A0 rv =3D self.handle_user_exception(e)
=C2=A0 File "/home/magnun/venv/pgadmin4/lib/python2.7/site-packages/flask/ap= p.py", line 1381, in handle_user_exception
=C2=A0=C2=A0=C2=A0 reraise(exc_type, exc_value, tb) =C2=A0 File "/home/magnun/venv/pgadmin4/lib/python2.7/site-packages/flask/ap= p.py", line 1475, in full_dispatch_request
=C2=A0=C2=A0=C2=A0 rv =3D self.dispatch_request()
=C2=A0 File "/home/magnun/venv/pgadmin4/lib/python2.7/site-packages/flask/ap= p.py", line 1461, in dispatch_request
=C2=A0=C2=A0=C2=A0 return self.view_functions[rule.endpoint](**req.view_args)
=C2=A0 File "/home/magnun/venv/pgadmin4/lib/python2.7/site-packages/flask_lo= gin.py", line 758, in decorated_view
=C2=A0=C2=A0=C2=A0 return func(*args, **kwargs)
=C2=A0 File "/home/magnun/venv/pgadmin4/lib/python2.7/site-packages/pgadmin4/pgadm= in/tools/sqleditor/__init__.py", line 241, in start_query_tool
=C2=A0=C2=A0=C2=A0 sql =3D json.loads(request.data.decode())
UnicodeDecodeError: 'ascii' codec can't decode byt= e 0xc3 in position 48: ordinal not in range(128)

But, the other way around isn't true. Executing the update mentione= d above in psql and then running a select in PgAdmin4 (select * from users where id=3D1) works fine.

I'll give it a try with Python 3 as soon as possible, it might solv= e all the issues since it uses unicode by default.
=
It's good to have feedback like above.
That will give us= the idea, where we need improvements, as all the current developers, worki= ng on pgAdmin 4, are from English background.

Than= ks for the inputs/feedbacks, please keep them coming.

<= p style=3D"margin:0pt">--

= Thanks & Regards,

Ashesh Vashi
EnterpriseDB INDIA:=C2=A0Enterprise PostgreSQL Company



Thanks in advance.

---
Magnun Leno


--001a1141ef5239fcf505377ba7a8--