Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtp (Exim 4.84_2) (envelope-from ) id 1bNHf7-00069m-Qx for pgadmin-hackers@arkaria.postgresql.org; Wed, 13 Jul 2016 10:45:06 +0000 Received: from localhost ([127.0.0.1] helo=postgresql.org) by malur.postgresql.org with smtp (Exim 4.84_2) (envelope-from ) id 1bNHf7-0003Ke-9r for pgadmin-hackers@arkaria.postgresql.org; Wed, 13 Jul 2016 10:45:05 +0000 Received: from makus.postgresql.org ([2001:4800:1501:1::229]) by malur.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1bN3Z9-0004FQ-PU for pgadmin-hackers@postgresql.org; Tue, 12 Jul 2016 19:42:00 +0000 Received: from mail-wm0-x232.google.com ([2a00:1450:400c:c09::232]) by makus.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.84_2) (envelope-from ) id 1bN3Z6-0000Cq-Js for pgadmin-hackers@postgresql.org; Tue, 12 Jul 2016 19:41:58 +0000 Received: by mail-wm0-x232.google.com with SMTP id f65so2928697wmi.0 for ; Tue, 12 Jul 2016 12:41:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=subject:to:references:from:cc:message-id:date:user-agent :mime-version:in-reply-to; bh=WlMjHT0Nl8rL/rdkZTWbY6a+F/KKx15x8z5OG2p/bDM=; b=quHlfhmtkTX351+Z2fNQuvwX16cwL9gtHWPn/x0qUoAg8wn+hznCKtlTRNfh5muQ8Q K8kwAEvBE4wTnABZxIQi5nE9XGGeg4wgIm29pykZWajNcWq6jKFgOSrNxQNMg6DUY/1O gfo9MR46VALVJFCdF6fXPIEIs6O6n8Z99ZmKIJil64gkPVcxb7E6GS3Bv7HQ6WF5ZhIe 7nsCggFoDWVJgvy0c9N0GUvqpkTY90MyHmwMU5QAcK09SM8EI+gbriiCdglJeIBf4aYD swTQ20Pbmpl1Spj9h/3xpzhe+D13ORdGyJltAHbAK562W/WbrIRwBamV6mMB5/ZFSGTl NfNQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:subject:to:references:from:cc:message-id:date :user-agent:mime-version:in-reply-to; bh=WlMjHT0Nl8rL/rdkZTWbY6a+F/KKx15x8z5OG2p/bDM=; b=a83Sa9kk9Ipvx0X50MkO3f9e54KJSStG3q8+4hLEk0kQmuCG0f5GwldUyehabHUKJl OtwzGBCfzLyDs6SPllJZsovVQZhpNtgJBRUDFSfxcM3aTLECQKdQL6y+/1FwCl58KQEs uyuDNs2NjAhDXnrgNwZ0vQa/9jPUGjEXZL5MguUtebP6S3W4/LhO/HsheqMGci3MqbaS yDSXhUDkYSv3YX+8d0KSMfwlFi/CuxesV0hskWiY8Jo5THoTLxMdKDNaTjhe7I/oVnzW copd4zMEKcOxN9c5rHUMBvmh13f7LQVdpkFs2cxl73pThdKUjxrd15bXNJaX9AD+8FB9 Edbg== X-Gm-Message-State: ALyK8tL/zY9dA9gXevZ8GwrQi/r+W5Zw6+oKszCf9KYPsTIu/tIwXuMJyEF1PgIBDzjMMw== X-Received: by 10.28.144.5 with SMTP id s5mr2501048wmd.39.1468352514656; Tue, 12 Jul 2016 12:41:54 -0700 (PDT) Received: from [0.0.0.0] (node1.tor.angristan.xyz. [163.172.135.172]) by smtp.gmail.com with ESMTPSA id q187sm9802820wma.17.2016.07.12.12.41.51 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 12 Jul 2016 12:41:53 -0700 (PDT) Subject: Re: [Patch] Fix Unicode in errmsg To: Ashesh Vashi References: <2048fa7e-7563-3edd-db2c-d58b97b98bee@gmail.com> From: Magnun Leno Cc: pgadmin-hackers@postgresql.org Message-ID: <5aef049f-d684-3a52-5b8f-0b31e3a74f32@gmail.com> Date: Tue, 12 Jul 2016 16:41:46 -0300 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.1.1 MIME-Version: 1.0 In-Reply-To: Content-Type: multipart/alternative; boundary="------------F330D6B12652B81262AD7718" X-Pg-Spam-Score: -2.7 (--) List-Archive: List-Help: List-ID: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: X-Mailing-List: pgadmin-hackers Precedence: bulk Sender: pgadmin-hackers-owner@postgresql.org This is a multi-part message in MIME format. --------------F330D6B12652B81262AD7718 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit > What error are you getting? > Can you please share the exact error message to understand the issue > properly? Sorry, my bad. Here is the traceback: 2016-07-12 16:12:22,366: ERROR pgadmin: Exception on /browser/server-group/obj/ [POST] Traceback (most recent call last): File "/home/magnun/venv/pgadmin4/lib/python2.7/site-packages/flask/app.py", line 1817, in wsgi_app response = self.full_dispatch_request() File "/home/magnun/venv/pgadmin4/lib/python2.7/site-packages/flask/app.py", line 1477, in full_dispatch_request rv = self.handle_user_exception(e) File "/home/magnun/venv/pgadmin4/lib/python2.7/site-packages/flask/app.py", line 1381, in handle_user_exception reraise(exc_type, exc_value, tb) File "/home/magnun/venv/pgadmin4/lib/python2.7/site-packages/flask/app.py", line 1475, in full_dispatch_request rv = self.dispatch_request() File "/home/magnun/venv/pgadmin4/lib/python2.7/site-packages/flask/app.py", line 1461, in dispatch_request return self.view_functions[rule.endpoint](**req.view_args) File "/home/magnun/venv/pgadmin4/lib/python2.7/site-packages/flask/views.py", line 84, in view return self.dispatch_request(*args, **kwargs) File "/home/magnun/venv/pgadmin4/lib/python2.7/site-packages/pgadmin4/pgadmin/browser/utils.py", line 235, in dispatch_request return method(*args, **kwargs) File "/home/magnun/venv/pgadmin4/lib/python2.7/site-packages/pgadmin4/pgadmin/browser/server_groups/__init__.py", line 204, in create data = request.form if request.form else json.loads(request.data.decode()) UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 14: ordinal not in range(128) How to reproduce: 1. Start PgAdmin4; 2. Access localhost:5050 3. Login 4. In the left panel, right click in Servers. Then, "Create > Server Group", or "Create > Server" (both share the same error) 5. In the "Name field", write any non ASCII word. In my case I was using the following name: Produção 6. Click "Save" More info about my env: $ env | grep LANG LANG=pt_BR.UTF-8 GDM_LANG=pt_BR.UTF-8 $ python --version Python 2.7.11 $ pip freeze Babel==1.3 beautifulsoup4==4.4.1 blinker==1.3 django-htmlmin==0.8.0 extras==0.0.3 fixtures==2.0.0 Flask==0.10.1 Flask-Babel==0.9 Flask-Gravatar==0.4.2 Flask-Login==0.2.11 Flask-Mail==0.9.1 Flask-Principal==0.4.0 Flask-Security==1.7.4 Flask-SQLAlchemy==2.0 Flask-WTF==0.11 html5lib==1.0b3 importlib==1.0.3 itsdangerous==0.24 Jinja2==2.7.3 linecache2==1.0.0 MarkupSafe==0.23 passlib==1.6.2 pbr==1.9.1 pgadmin4==1.0b2 psycopg2==2.5.2 pycrypto==2.6.1 pyrsistent==0.11.13 python-dateutil==2.5.0 python-mimeparse==1.5.1 pytz==2014.10 simplejson==3.6.5 six==1.9.0 speaklater==1.3 SQLAlchemy==0.9.8 sqlparse==0.1.19 testscenarios==0.5.0 testtools==2.0.0 traceback2==1.4.0 unittest2==1.1.0 Werkzeug==0.9.6 WTForms==2.0.2 > > We will need to change too many places by this logic. > I've attached patch for it. I'm not familiar with the code base yet, but not necessarily all occurrences of "decode()" need to be patched, just the ones that deal with user input. I've done some further testing and noticed that even running an insert or update with non ASCII text runs into failure. This "update users set cidade='Brasília' where id=1;", results in the following traceback: 2016-07-12 16:32:59,964: ERROR pgadmin: Exception on /sqleditor/query_tool/start/2760583 [POST] Traceback (most recent call last): File "/home/magnun/venv/pgadmin4/lib/python2.7/site-packages/flask/app.py", line 1817, in wsgi_app response = self.full_dispatch_request() File "/home/magnun/venv/pgadmin4/lib/python2.7/site-packages/flask/app.py", line 1477, in full_dispatch_request rv = self.handle_user_exception(e) File "/home/magnun/venv/pgadmin4/lib/python2.7/site-packages/flask/app.py", line 1381, in handle_user_exception reraise(exc_type, exc_value, tb) File "/home/magnun/venv/pgadmin4/lib/python2.7/site-packages/flask/app.py", line 1475, in full_dispatch_request rv = self.dispatch_request() File "/home/magnun/venv/pgadmin4/lib/python2.7/site-packages/flask/app.py", line 1461, in dispatch_request return self.view_functions[rule.endpoint](**req.view_args) File "/home/magnun/venv/pgadmin4/lib/python2.7/site-packages/flask_login.py", line 758, in decorated_view return func(*args, **kwargs) File "/home/magnun/venv/pgadmin4/lib/python2.7/site-packages/pgadmin4/pgadmin/tools/sqleditor/__init__.py", line 241, in start_query_tool sql = json.loads(request.data.decode()) UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 48: ordinal not in range(128) But, the other way around isn't true. Executing the update mentioned above in psql and then running a select in PgAdmin4 (select * from users where id=1) works fine. I'll give it a try with Python 3 as soon as possible, it might solve all the issues since it uses unicode by default. Thanks in advance. --- Magnun Leno --------------F330D6B12652B81262AD7718 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit
What error are you getting?
Can you please share the exact error message to understand the issue properly?
Sorry, my bad. Here is the traceback:

2016-07-12 16:12:22,366: ERROR    pgadmin:    Exception on /browser/server-group/obj/ [POST]
Traceback (most recent call last):
  File "/home/magnun/venv/pgadmin4/lib/python2.7/site-packages/flask/app.py", line 1817, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/magnun/venv/pgadmin4/lib/python2.7/site-packages/flask/app.py", line 1477, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/magnun/venv/pgadmin4/lib/python2.7/site-packages/flask/app.py", line 1381, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/home/magnun/venv/pgadmin4/lib/python2.7/site-packages/flask/app.py", line 1475, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/magnun/venv/pgadmin4/lib/python2.7/site-packages/flask/app.py", line 1461, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/home/magnun/venv/pgadmin4/lib/python2.7/site-packages/flask/views.py", line 84, in view
    return self.dispatch_request(*args, **kwargs)
  File "/home/magnun/venv/pgadmin4/lib/python2.7/site-packages/pgadmin4/pgadmin/browser/utils.py", line 235, in dispatch_request
    return method(*args, **kwargs)
  File "/home/magnun/venv/pgadmin4/lib/python2.7/site-packages/pgadmin4/pgadmin/browser/server_groups/__init__.py", line 204, in create
    data = request.form if request.form else json.loads(request.data.decode())
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 14: ordinal not in range(128)

How to reproduce:
  1. Start PgAdmin4;
  2. Access localhost:5050
  3. Login
  4. In the left panel, right click in Servers. Then, "Create > Server Group", or "Create > Server" (both share the same error)
  5. In the "Name field", write any non ASCII word. In my case I was using the following name: Produção
  6. Click "Save"
More info about my env:

$ env | grep LANG
LANG=pt_BR.UTF-8
GDM_LANG=pt_BR.UTF-8

$ python --version
Python 2.7.11

$ pip freeze
Babel==1.3
beautifulsoup4==4.4.1
blinker==1.3
django-htmlmin==0.8.0
extras==0.0.3
fixtures==2.0.0
Flask==0.10.1
Flask-Babel==0.9
Flask-Gravatar==0.4.2
Flask-Login==0.2.11
Flask-Mail==0.9.1
Flask-Principal==0.4.0
Flask-Security==1.7.4
Flask-SQLAlchemy==2.0
Flask-WTF==0.11
html5lib==1.0b3
importlib==1.0.3
itsdangerous==0.24
Jinja2==2.7.3
linecache2==1.0.0
MarkupSafe==0.23
passlib==1.6.2
pbr==1.9.1
pgadmin4==1.0b2
psycopg2==2.5.2
pycrypto==2.6.1
pyrsistent==0.11.13
python-dateutil==2.5.0
python-mimeparse==1.5.1
pytz==2014.10
simplejson==3.6.5
six==1.9.0
speaklater==1.3
SQLAlchemy==0.9.8
sqlparse==0.1.19
testscenarios==0.5.0
testtools==2.0.0
traceback2==1.4.0
unittest2==1.1.0
Werkzeug==0.9.6
WTForms==2.0.2


We will need to change too many places by this logic.
I've attached patch for it.
I'm not familiar with the code base yet, but not necessarily all occurrences of "decode()" need to be patched, just the ones that deal with user input.

I've done some further testing and noticed that even running an insert or update with non ASCII text runs into failure. This "update users set cidade='Brasília' where id=1;", results in the following traceback:

2016-07-12 16:32:59,964: ERROR    pgadmin:    Exception on /sqleditor/query_tool/start/2760583 [POST]
Traceback (most recent call last):
  File "/home/magnun/venv/pgadmin4/lib/python2.7/site-packages/flask/app.py", line 1817, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/magnun/venv/pgadmin4/lib/python2.7/site-packages/flask/app.py", line 1477, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/magnun/venv/pgadmin4/lib/python2.7/site-packages/flask/app.py", line 1381, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/home/magnun/venv/pgadmin4/lib/python2.7/site-packages/flask/app.py", line 1475, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/magnun/venv/pgadmin4/lib/python2.7/site-packages/flask/app.py", line 1461, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/home/magnun/venv/pgadmin4/lib/python2.7/site-packages/flask_login.py", line 758, in decorated_view
    return func(*args, **kwargs)
  File "/home/magnun/venv/pgadmin4/lib/python2.7/site-packages/pgadmin4/pgadmin/tools/sqleditor/__init__.py", line 241, in start_query_tool
    sql = json.loads(request.data.decode())
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 48: ordinal not in range(128)

But, the other way around isn't true. Executing the update mentioned above in psql and then running a select in PgAdmin4 (select * from users where id=1) works fine.

I'll give it a try with Python 3 as soon as possible, it might solve all the issues since it uses unicode by default.

Thanks in advance.

---
Magnun Leno

--------------F330D6B12652B81262AD7718--