Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtp (Exim 4.84_2) (envelope-from ) id 1bwpiR-0002p2-8w for pgadmin-hackers@arkaria.postgresql.org; Wed, 19 Oct 2016 12:11:27 +0000 Received: from localhost ([127.0.0.1] helo=postgresql.org) by malur.postgresql.org with smtp (Exim 4.84_2) (envelope-from ) id 1bwpiQ-0003yP-Rj for pgadmin-hackers@arkaria.postgresql.org; Wed, 19 Oct 2016 12:11:26 +0000 Received: from makus.postgresql.org ([2001:4800:1501:1::229]) by malur.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1bwpiO-0003xY-Cw for pgadmin-hackers@postgresql.org; Wed, 19 Oct 2016 12:11:25 +0000 Received: from mail-lf0-x230.google.com ([2a00:1450:4010:c07::230]) by makus.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.84_2) (envelope-from ) id 1bwpiK-00073C-3L for pgadmin-hackers@postgresql.org; Wed, 19 Oct 2016 12:11:23 +0000 Received: by mail-lf0-x230.google.com with SMTP id b81so21267406lfe.1 for ; Wed, 19 Oct 2016 05:11:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=enterprisedb-com.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=6RURVBAzt2MRK8Ra1rMU7Gwagjew5VtyQutcbK+vXd0=; b=PoHh7lX1rDB4MxzkbN5GFnVg1cQ1uAHAFFkd3c4vtvsaVMFZJKfBYK9ib/bk/GQ4Pi dMoZabe7PWtfdcAdt053v8XJhh5KWjVNHohhSqNlCNvZPlRn1LFzp5HcCjKL6aqvqEOS xpZunHaa1aCzwJWJrJyTejnmaJUZ0fXmRhurOTbzg/ahc+qD4+xulcsRME5FEc5StFhV 37U+3yyrde3wrgkmPoixf9oa3/jrWEUWNayqrQ8haIDoEeX77ppNbYQXfpbykTOZg/Q/ MfZtU7Rf+43RD1Ck2zHpxO9zP1Kx9/HsRaUEh0qABJwH4xOcKBqlYc57J3KWsBrl/Z5Y TAGw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=6RURVBAzt2MRK8Ra1rMU7Gwagjew5VtyQutcbK+vXd0=; b=Wr+8srpLvk0jtWfdrTQle7WSOyp6RLofR+ikKMkzO3Rht9foFCl5w0MNUJ7tUQBoLT 5FwWLgbSI/pCTPSXy0oq+sl4zMWdMJZWfGje6LmjKudAm0Lv1aLK34nQMR9SBjliiweB WuULvdJOKcTHNu76Cn+cFRV7Ljs3rvzUnTkuHOJ7h72YjjdREiA3kco/RU9fK4TNS9+V aRTIkjs37Jn7KQUQt9UKcWnT2KlZYchp4PmjCCk0RAkawx7vigRBd6OUAi29E04fAY8l asmqwCmbhQl79f05JlXR81tRNTBoRJ5Tfmj2NN8UXzGB+isbbGCGbOfzVZrlWdgxyNAw 4PVg== X-Gm-Message-State: AA6/9Rm9OHfYpzxzEUYWIOa9YPdlRF7XLJtFhvzorJRwHIK0IWpiGcG+qwoW+/Kbz25efcJkRVtdBPlFpEOHqlkQ X-Received: by 10.28.38.195 with SMTP id m186mr4706504wmm.81.1476879076638; Wed, 19 Oct 2016 05:11:16 -0700 (PDT) MIME-Version: 1.0 Received: by 10.80.135.227 with HTTP; Wed, 19 Oct 2016 05:11:15 -0700 (PDT) In-Reply-To: References: From: Neel Patel Date: Wed, 19 Oct 2016 17:41:15 +0530 Message-ID: Subject: Re: RM1849: Auto-generating security keys To: Fahar Abbas Cc: Ashesh Vashi , Dave Page , pgadmin-hackers , Josh Berkus , =?UTF-8?B?RGV2cmltIEfDnE5Ew5xa?= , Magnus Hagander , Sandeep Thakkar , Hamid Quddus Akhtar Content-Type: multipart/alternative; boundary=94eb2c03fbc2bd6115053f36b42d X-Pg-Spam-Score: -2.6 (--) List-Archive: List-Help: List-ID: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: X-Mailing-List: pgadmin-hackers Precedence: bulk Sender: pgadmin-hackers-owner@postgresql.org --94eb2c03fbc2bd6115053f36b42d Content-Type: text/plain; charset=UTF-8 Hi, Just to update for Python 3. It gives below error while running "pgAdmin4.py". ##### Traceback (most recent call last): File "/usr/lib/python3.4/threading.py", line 920, in _bootstrap_inner self.run() File "/usr/lib/python3.4/threading.py", line 868, in run self._target(*self._args, **self._kwargs) File "/usr/lib/python3.4/socketserver.py", line 620, in process_request_thread self.handle_error(request, client_address) File "/usr/lib/python3.4/socketserver.py", line 617, in process_request_thread self.finish_request(request, client_address) File "/usr/lib/python3.4/socketserver.py", line 344, in finish_request self.RequestHandlerClass(request, client_address, self) File "/usr/lib/python3.4/socketserver.py", line 673, in __init__ self.handle() File "/home/neel/workspace/pgAdmin4_3_4/lib/python3.4/site-packages/werkzeug/serving.py", line 200, in handle rv = BaseHTTPRequestHandler.handle(self) File "/usr/lib/python3.4/http/server.py", line 398, in handle self.handle_one_request() File "/home/neel/workspace/pgAdmin4_3_4/lib/python3.4/site-packages/werkzeug/serving.py", line 235, in handle_one_request return self.run_wsgi() File "/home/neel/workspace/pgAdmin4_3_4/lib/python3.4/site-packages/werkzeug/serving.py", line 177, in run_wsgi execute(self.server.app) File "/home/neel/workspace/pgAdmin4_3_4/lib/python3.4/site-packages/werkzeug/serving.py", line 165, in execute application_iter = app(environ, start_response) File "/home/neel/workspace/pgAdmin4_3_4/lib/python3.4/site-packages/flask/app.py", line 2000, in __call__ return self.wsgi_app(environ, start_response) File "/home/neel/workspace/pgAdmin4_3_4/lib/python3.4/site-packages/flask/app.py", line 1991, in wsgi_app response = self.make_response(self.handle_exception(e)) File "/home/neel/workspace/pgAdmin4_3_4/lib/python3.4/site-packages/flask/app.py", line 1567, in handle_exception reraise(exc_type, exc_value, tb) File "/home/neel/workspace/pgAdmin4_3_4/lib/python3.4/site-packages/flask/_compat.py", line 33, in reraise raise value File "/home/neel/workspace/pgAdmin4_3_4/lib/python3.4/site-packages/flask/app.py", line 1988, in wsgi_app response = self.full_dispatch_request() File "/home/neel/workspace/pgAdmin4_3_4/lib/python3.4/site-packages/flask/app.py", line 1643, in full_dispatch_request response = self.process_response(response) File "/home/neel/workspace/pgAdmin4_3_4/lib/python3.4/site-packages/flask/app.py", line 1864, in process_response self.save_session(ctx.session, response) File "/home/neel/workspace/pgAdmin4_3_4/lib/python3.4/site-packages/flask/app.py", line 926, in save_session return self.session_interface.save_session(self, session, response) File "/home/neel/Projects/pgAdmin4/pgadmin4_patch/pgadmin4/web/pgadmin/utils/session.py", line 267, in save_session self.manager.put(session) File "/home/neel/Projects/pgAdmin4/pgadmin4_patch/pgadmin4/web/pgadmin/utils/session.py", line 144, in put self.parent.put(session) File "/home/neel/Projects/pgAdmin4/pgadmin4_patch/pgadmin4/web/pgadmin/utils/session.py", line 214, in put session.sign(self.secret) File "/home/neel/Projects/pgAdmin4/pgadmin4_patch/pgadmin4/web/pgadmin/utils/session.py", line 71, in sign self.hmac_digest = _calc_hmac('%s:%s' % (self.sid, self.randval), secret) File "/home/neel/Projects/pgAdmin4/pgadmin4_patch/pgadmin4/web/pgadmin/utils/session.py", line 44, in _calc_hmac secret.encode(), body.encode(), hashlib.sha1 AttributeError: 'bytes' object has no attribute 'encode' ####### Thanks, Neel Patel On Wed, Oct 19, 2016 at 5:12 PM, Fahar Abbas wrote: > > > On Wed, Oct 19, 2016 at 4:03 PM, Fahar Abbas > wrote: > >> >> >> On Wed, Oct 19, 2016 at 3:55 PM, Ashesh Vashi < >> ashesh.vashi@enterprisedb.com> wrote: >> >>> Hi Fahar, >>> >>> Please log the case on redmine. >>> >> https://redmine.postgresql.org/issues/1871 >> >>> Please find the attached patch, please apply it locally, and test it. >>> >>> And, please update the case, and this mail chain accordingly. >>> >> This is resolved now and no error message displayed when we apply the > patch that is already shared. > >> >>> Sure Will test the patch and update the status accordingly. >> >>> -- >>> >>> Thanks & Regards, >>> >>> Ashesh Vashi >>> EnterpriseDB INDIA: Enterprise PostgreSQL Company >>> >>> >>> >>> *http://www.linkedin.com/in/asheshvashi* >>> >>> >>> On Wed, Oct 19, 2016 at 3:47 PM, Fahar Abbas < >>> fahar.abbas@enterprisedb.com> wrote: >>> >>>> Here is the output of if we copy config_local.py and execute python >>>> setup.py >>>> pgAdmin 4 - Application Initialisation >>>> ====================================== >>>> >>>> >>>> The configuration database - '/home/fahar/.pgadmin/pgadmin4.db' does >>>> not exist. >>>> Entering initial setup mode... >>>> NOTE: Configuring authentication for SERVER mode. >>>> >>>> >>>> Enter the email address and password to use for the initial pgAdmin >>>> user account: >>>> >>>> Email address: fahar.abbas@enterprisedb.com >>>> Password: >>>> Retype password: >>>> Traceback (most recent call last): >>>> File "setup.py", line 449, in >>>> do_setup(app) >>>> File "setup.py", line 96, in do_setup >>>> password = encrypt_password(p1) >>>> File "/home/fahar/venv/lib/python3.5/site-packages/flask_security/utils.py", >>>> line 150, in encrypt_password >>>> signed = get_hmac(password).decode('ascii') >>>> File "/home/fahar/venv/lib/python3.5/site-packages/flask_security/utils.py", >>>> line 108, in get_hmac >>>> 'set to "%s"' % _security.password_hash) >>>> RuntimeError: The configuration value `SECURITY_PASSWORD_SALT` must not >>>> be None when the value of `SECURITY_PASSWORD_HASH` is set to "pbkdf2_sha512" >>>> python setup.py >>>> pgAdmin 4 - Application Initialisation >>>> ====================================== >>>> >>>> User can not do any setup for web based now. >>>> >>>> >>>> The configuration database - '/home/fahar/.pgadmin/pgadmin4.db' does >>>> not exist. >>>> Entering initial setup mode... >>>> NOTE: Configuring authentication for SERVER mode. >>>> >>>> >>>> Enter the email address and password to use for the initial pgAdmin >>>> user account: >>>> >>>> Email address: fahar.abbas@enterprisedb.com >>>> Password: >>>> Retype password: >>>> Traceback (most recent call last): >>>> File "setup.py", line 449, in >>>> do_setup(app) >>>> File "setup.py", line 96, in do_setup >>>> password = encrypt_password(p1) >>>> File "/home/fahar/venv/lib/python3.5/site-packages/flask_security/utils.py", >>>> line 150, in encrypt_password >>>> signed = get_hmac(password).decode('ascii') >>>> File "/home/fahar/venv/lib/python3.5/site-packages/flask_security/utils.py", >>>> line 108, in get_hmac >>>> 'set to "%s"' % _security.password_hash) >>>> RuntimeError: The configuration value `SECURITY_PASSWORD_SALT` must not >>>> be None when the value of `SECURITY_PASSWORD_HASH` is set to "pbkdf2_sha512" >>>> >>>> On Wed, Oct 19, 2016 at 3:03 PM, Fahar Abbas < >>>> fahar.abbas@enterprisedb.com> wrote: >>>> >>>>> Dave, >>>>> >>>>> Testing Environment >>>>> >>>>> Ubuntu 16.04 Linux 64: >>>>> -------------------------------- >>>>> >>>>> pg-AdminIV Development Environment Setup for Ubuntu : >>>>> >>>>> >>>>> 1) Install GIT >>>>> >>>>> sudo apt-get install git >>>>> >>>>> 2) Install pip3 >>>>> >>>>> sudo apt-get install python3-pip >>>>> >>>>> 3) Install virtualenv >>>>> >>>>> sudo pip3 install virtualenv >>>>> >>>>> 4) install below dependency as it is required for psycopg2 & pycrypto >>>>> module >>>>> >>>>> sudo apt-get install libpq-dev >>>>> >>>>> sudo apt-get install python3-dev >>>>> >>>>> 5) Create virtual environment >>>>> >>>>> virtualenv -p python3 venv >>>>> >>>>> 6) Create mkdir Projects >>>>> >>>>> 7) Clone git repo in Projects >>>>> >>>>> git clone http://git.postgresql.org/git/pgadmin4.git >>>>> >>>>> 8) activate virtual environment >>>>> >>>>> source venv/bin/activate >>>>> >>>>> 9) Install modules >>>>> >>>>> pip3 install -r requirements_py3.txt >>>>> >>>>> *10) Edit the config.py file to config_local.py resides in >>>>> Projects\pgAdmin4\web * >>>>> >>>>> 11)Now run setup.py file (\Projects\pgAdmin4\web) >>>>> python setup.py >>>>> >>>>> If user does not create config_local.py and do Python setup.py for new >>>>> Development then SECURITY_PASSWORD_SALT message is also displayed: >>>>> >>>>> Here is the output: >>>>> ------------------------- >>>>> >>>>> python setup.py >>>>> pgAdmin 4 - Application Initialisation >>>>> ====================================== >>>>> >>>>> >>>>> The configuration database - '/home/fahar/.pgadmin/pgadmin4.db' does >>>>> not exist. >>>>> Entering initial setup mode... >>>>> NOTE: Configuring authentication for SERVER mode. >>>>> >>>>> >>>>> Enter the email address and password to use for the initial >>>>> pgAdmin user account: >>>>> >>>>> Email address: fahar.abbas@enterprisedb.com >>>>> Password: >>>>> Retype password: >>>>> Traceback (most recent call last): >>>>> File "setup.py", line 449, in >>>>> do_setup(app) >>>>> File "setup.py", line 96, in do_setup >>>>> password = encrypt_password(p1) >>>>> File "/home/fahar/venv/lib/python3.5/site-packages/flask_security/utils.py", >>>>> line 150, in encrypt_password >>>>> signed = get_hmac(password).decode('ascii') >>>>> File "/home/fahar/venv/lib/python3.5/site-packages/flask_security/utils.py", >>>>> line 108, in get_hmac >>>>> 'set to "%s"' % _security.password_hash) >>>>> RuntimeError: The configuration value `SECURITY_PASSWORD_SALT` must >>>>> not be None when the value of `SECURITY_PASSWORD_HASH` is set to >>>>> "pbkdf2_sha512" >>>>> (venv) fahar@fahar-virtual-machine:~/Projects/pgadmin4/web$ >>>>> >>>>> >>>>> Is this expected? >>>>> >>>>> On Wed, Oct 19, 2016 at 1:37 PM, Fahar Abbas < >>>>> fahar.abbas@enterprisedb.com> wrote: >>>>> >>>>>> Sure, >>>>>> >>>>>> Will test this thoroughly after complete investigation. >>>>>> >>>>>> Kind Regards, >>>>>> >>>>>> On Wed, Oct 19, 2016 at 1:27 PM, Dave Page wrote: >>>>>> >>>>>>> Patch applied. >>>>>>> >>>>>>> Fahar, can you please test this thoroughly in desktop and server >>>>>>> modes, with both fresh and upgraded installations? >>>>>>> >>>>>>> https://redmine.postgresql.org/issues/1849 >>>>>>> >>>>>>> Packagers: This change means that packages are no longer forced to >>>>>>> create a config_local.py file, and there is no longer any need to >>>>>>> explicitly set SECURITY_PASSWORD_SALT, SECURITY_KEY >>>>>>> and CSRF_SESSION_KEY in the config (in fact, they should be removed for new >>>>>>> installations, if you have included them in 1.0) >>>>>>> >>>>>>> Thanks. >>>>>>> >>>>>>> >>>>>>> On Wed, Oct 19, 2016 at 6:46 AM, Ashesh Vashi < >>>>>>> ashesh.vashi@enterprisedb.com> wrote: >>>>>>> >>>>>>>> Hi Dave, >>>>>>>> >>>>>>>> On Sat, Oct 15, 2016 at 8:02 AM, Dave Page >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hi >>>>>>>>> >>>>>>>>> >>>>>>>>> On Friday, October 14, 2016, Dave Page wrote: >>>>>>>>> >>>>>>>>>> Hi >>>>>>>>>> >>>>>>>>>> On Thursday, October 13, 2016, Ashesh Vashi < >>>>>>>>>> ashesh.vashi@enterprisedb.com> wrote: >>>>>>>>>> >>>>>>>>>>> Hi Dave, >>>>>>>>>>> >>>>>>>>>>> On Tue, Oct 11, 2016 at 9:10 PM, Dave Page >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi Ashesh, >>>>>>>>>>>> >>>>>>>>>>>> Can you please review the attached patch, and apply if you're >>>>>>>>>>>> happy with it? >>>>>>>>>>>> >>>>>>>>>>> Overall the patch looked good to me. >>>>>>>>>>> But - I encounter an issue in 'web' mode, which wont happen with >>>>>>>>>>> 'runtime'. >>>>>>>>>>> >>>>>>>>>>> Steps for reproduction on existing pgAdmin 4 environment with >>>>>>>>>>> 'web' mode. >>>>>>>>>>> - Apply the patch >>>>>>>>>>> - Start the pgAdmin4 application (stand alone application). >>>>>>>>>>> - Open pgAdmin home page. >>>>>>>>>>> - Log out (if already login). >>>>>>>>>>> >>>>>>>>>>> And, you will see an exception. >>>>>>>>>>> >>>>>>>>>>> I have figure out the issue with the patch. >>>>>>>>>>> We were setting the SECURITY_PASSWORD_SALT, after initializing >>>>>>>>>>> the Security object. >>>>>>>>>>> Hence - it could not set the SECURITY_KEY, and >>>>>>>>>>> SECURITY_PASSWORD_SALT properly. >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Hmm. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> I had moved the Security object initialization after fetching >>>>>>>>>>> these configurations from the database. >>>>>>>>>>> I have attached a addon patch for the same. >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> OK, thanks. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Now - I run into another issue. >>>>>>>>>>> Because - the existing password was hashed using the old >>>>>>>>>>> SECURITY_PASSWORD_SALT, I am no more able to login to pgAdmin 4. >>>>>>>>>>> >>>>>>>>>>> I think - we need to think about different strategy for >>>>>>>>>>> upgrading the configuration file in the 'web' mode. >>>>>>>>>>> I was thinking - we can store the existing security >>>>>>>>>>> configurations in the database during upgrade process in 'web' mode. >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> My concern with that is that we'll likely be storing the default >>>>>>>>>> config values in many cases, thus for those users, perpetuating the problem. >>>>>>>>>> >>>>>>>>>> I guess what we need to do is re-encrypt the password during the >>>>>>>>>> upgrade - however, that makes me think; we then have both the key and the >>>>>>>>>> encrypted passwords in the same database which is clearly not a good idea. >>>>>>>>>> Sigh... Needs more thought. >>>>>>>>>> >>>>>>>>> >>>>>>>>> OK, so I've been thinking about this and experimenting for a >>>>>>>>> couple of hours, as well as annoying the crap out of Magnus by thinking out >>>>>>>>> loud in his general direction, and it looks like this isn't a major problem >>>>>>>>> as from what I can see, SECURITY_PASSWORD_SALT is (aside from really being >>>>>>>>> a key not a salt) not the only salting that's done. >>>>>>>>> >>>>>>>>> It looks like it's used system-wide as the key to generate an HMAC >>>>>>>>> of the users password, which is then passed to passlib which salts and >>>>>>>>> hashes it. I did some testing, and found that two users with the same >>>>>>>>> password end up with different hashes in the database, so clearly there is >>>>>>>>> also per-user salting happening. I also created two users, then dropped the >>>>>>>>> database and created the same user accounts with the same passwords again, >>>>>>>>> and found that the resulting hashes were different in both databases - thus >>>>>>>>> there is something else ensuring the hashes are unique across different >>>>>>>>> installations/databases. >>>>>>>>> >>>>>>>>> So, I believe we can do as you suggest and migrate existing values >>>>>>>>> for SECURITY_PASSWORD_SALT, given that there's clearly some other per user >>>>>>>>> and per installation/database salting going on anyway. New installations >>>>>>>>> can have the random value for SECURITY_PASSWORD_SALT. >>>>>>>>> >>>>>>>> We do not need to generate the random SECURITY_PASSWORD_SALT during >>>>>>>> upgrade mode, which was wrong added in my addon patch. >>>>>>>> >>>>>>>> Please find the updated patch. >>>>>>>> >>>>>>>> Otherwise - looks good to me. >>>>>>>> Please commit the new patch (if you're ok with the change). >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> >>>>>>>> Thanks & Regards, >>>>>>>> >>>>>>>> Ashesh Vashi >>>>>>>> EnterpriseDB INDIA: Enterprise PostgreSQL Company >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> *http://www.linkedin.com/in/asheshvashi* >>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> I don't believe SECURITY_KEY and CSRF_SESSION_KEY are issues >>>>>>>>> either, as they're used for purposes that are essentially ephemeral, and >>>>>>>>> thus can be changed during an upgrade. >>>>>>>>> >>>>>>>>> Adding Magnus as I'd appreciate any thoughts he may have. >>>>>>>>> >>>>>>>>> Patch attached - please review (Ashesh, but others too would be >>>>>>>>> appreciated)! >>>>>>>>> >>>>>>>>> Thanks. >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Dave Page >>>>>>>>> Blog: http://pgsnake.blogspot.com >>>>>>>>> Twitter: @pgsnake >>>>>>>>> >>>>>>>>> EnterpriseDB UK: http://www.enterprisedb.com >>>>>>>>> The Enterprise PostgreSQL Company >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Dave Page >>>>>>> Blog: http://pgsnake.blogspot.com >>>>>>> Twitter: @pgsnake >>>>>>> >>>>>>> EnterpriseDB UK: http://www.enterprisedb.com >>>>>>> The Enterprise PostgreSQL Company >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Syed Fahar Abbas >>>>>> Quality Management Group >>>>>> >>>>>> EnterpriseDB Corporation >>>>>> Phone Office: +92-51-835-8874 >>>>>> Phone Direct: +92-51-8466803 >>>>>> Mobile: +92-333-5409707 >>>>>> Skype ID: syed.fahar.abbas >>>>>> Website: www.enterprisedb.com >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Syed Fahar Abbas >>>>> Quality Management Group >>>>> >>>>> EnterpriseDB Corporation >>>>> Phone Office: +92-51-835-8874 >>>>> Phone Direct: +92-51-8466803 >>>>> Mobile: +92-333-5409707 >>>>> Skype ID: syed.fahar.abbas >>>>> Website: www.enterprisedb.com >>>>> >>>> >>>> >>>> >>>> -- >>>> Syed Fahar Abbas >>>> Quality Management Group >>>> >>>> EnterpriseDB Corporation >>>> Phone Office: +92-51-835-8874 >>>> Phone Direct: +92-51-8466803 >>>> Mobile: +92-333-5409707 >>>> Skype ID: syed.fahar.abbas >>>> Website: www.enterprisedb.com >>>> >>> >>> >> >> >> -- >> Syed Fahar Abbas >> Quality Management Group >> >> EnterpriseDB Corporation >> Phone Office: +92-51-835-8874 >> Phone Direct: +92-51-8466803 >> Mobile: +92-333-5409707 >> Skype ID: syed.fahar.abbas >> Website: www.enterprisedb.com >> > > > > -- > Syed Fahar Abbas > Quality Management Group > > EnterpriseDB Corporation > Phone Office: +92-51-835-8874 > Phone Direct: +92-51-8466803 > Mobile: +92-333-5409707 > Skype ID: syed.fahar.abbas > Website: www.enterprisedb.com > --94eb2c03fbc2bd6115053f36b42d Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hi,

Just to update for Python 3.
<= div>It gives below error while running "pgAdmin4.py".
<= br>
#####

Traceback (most recent ca= ll last):
=C2=A0 File "/usr/lib/python3.4/threading.py"= , line 920, in _bootstrap_inner
=C2=A0 =C2=A0 self.run()
=C2=A0 File "/usr/lib/python3.4/threading.py", line 868, in run=
=C2=A0 =C2=A0 self._target(*self._args, **self._kwargs)
=C2=A0 File "/usr/lib/python3.4/socketserver.py", line 620, in = process_request_thread
=C2=A0 =C2=A0 self.handle_error(request, c= lient_address)
=C2=A0 File "/usr/lib/python3.4/socketserver.= py", line 617, in process_request_thread
=C2=A0 =C2=A0 self.= finish_request(request, client_address)
=C2=A0 File "/usr/li= b/python3.4/socketserver.py", line 344, in finish_request
= =C2=A0 =C2=A0 self.RequestHandlerClass(request, client_address, self)
=
=C2=A0 File "/usr/lib/python3.4/socketserver.py", line 673, = in __init__
=C2=A0 =C2=A0 self.handle()
=C2=A0 File &qu= ot;/home/neel/workspace/pgAdmin4_3_4/lib/python3.4/site-packages/werkzeug/s= erving.py", line 200, in handle
=C2=A0 =C2=A0 rv =3D BaseHTT= PRequestHandler.handle(self)
=C2=A0 File "/usr/lib/python3.4= /http/server.py", line 398, in handle
=C2=A0 =C2=A0 self.han= dle_one_request()
=C2=A0 File "/home/neel/workspace/pgAdmin4= _3_4/lib/python3.4/site-packages/werkzeug/serving.py", line 235, in ha= ndle_one_request
=C2=A0 =C2=A0 return self.run_wsgi()
= =C2=A0 File "/home/neel/workspace/pgAdmin4_3_4/lib/python3.4/site-pack= ages/werkzeug/serving.py", line 177, in run_wsgi
=C2=A0 =C2= =A0 execute(self.server.app)
=C2=A0 File "/home/neel/workspa= ce/pgAdmin4_3_4/lib/python3.4/site-packages/werkzeug/serving.py", line= 165, in execute
=C2=A0 =C2=A0 application_iter =3D app(environ, = start_response)
=C2=A0 File "/home/neel/workspace/pgAdmin4_3= _4/lib/python3.4/site-packages/flask/app.py", line 2000, in __call__
=C2=A0 =C2=A0 return self.wsgi_app(environ, start_response)
<= div>=C2=A0 File "/home/neel/workspace/pgAdmin4_3_4/lib/python3.4/site-= packages/flask/app.py", line 1991, in wsgi_app
=C2=A0 =C2=A0= response =3D self.make_response(self.handle_exception(e))
=C2=A0= File "/home/neel/workspace/pgAdmin4_3_4/lib/python3.4/site-packages/f= lask/app.py", line 1567, in handle_exception
=C2=A0 =C2=A0 r= eraise(exc_type, exc_value, tb)
=C2=A0 File "/home/neel/work= space/pgAdmin4_3_4/lib/python3.4/site-packages/flask/_compat.py", line= 33, in reraise
=C2=A0 =C2=A0 raise value
=C2=A0 File &= quot;/home/neel/workspace/pgAdmin4_3_4/lib/python3.4/site-packages/flask/ap= p.py", line 1988, in wsgi_app
=C2=A0 =C2=A0 response =3D sel= f.full_dispatch_request()
=C2=A0 File "/home/neel/workspace/= pgAdmin4_3_4/lib/python3.4/site-packages/flask/app.py", line 1643, in = full_dispatch_request
=C2=A0 =C2=A0 response =3D self.process_res= ponse(response)
=C2=A0 File "/home/neel/workspace/pgAdmin4_3= _4/lib/python3.4/site-packages/flask/app.py", line 1864, in process_re= sponse
=C2=A0 =C2=A0 self.save_session(ctx.session, response)
=C2=A0 File "/home/neel/workspace/pgAdmin4_3_4/lib/python3.4/si= te-packages/flask/app.py", line 926, in save_session
=C2=A0 = =C2=A0 return self.session_interface.save_session(self, session, response)<= /div>
=C2=A0 File "/home/neel/Projects/pgAdmin4/pgadmin4_patch/pga= dmin4/web/pgadmin/utils/session.py", line 267, in save_session
=C2=A0 =C2=A0 self.manager.put(session)
=C2=A0 File "/hom= e/neel/Projects/pgAdmin4/pgadmin4_patch/pgadmin4/web/pgadmin/utils/session.= py", line 144, in put
=C2=A0 =C2=A0 self.parent.put(session)=
=C2=A0 File "/home/neel/Projects/pgAdmin4/pgadmin4_patch/pg= admin4/web/pgadmin/utils/session.py", line 214, in put
=C2= =A0 =C2=A0 session.sign(self.secret)
=C2=A0 File "/home/neel= /Projects/pgAdmin4/pgadmin4_patch/pgadmin4/web/pgadmin/utils/session.py&quo= t;, line 71, in sign
=C2=A0 =C2=A0 self.hmac_digest =3D _calc_hma= c('%s:%s' % (self.sid, self.randval), secret)
=C2=A0 File= "/home/neel/Projects/pgAdmin4/pgadmin4_patch/pgadmin4/web/pgadmin/uti= ls/session.py", line 44, in _calc_hmac
=C2=A0 =C2=A0 secret.= encode(), body.encode(), hashlib.sha1
AttributeError: 'bytes&= #39; object has no attribute 'encode'
=C2=A0#######=

Thanks,
Neel Patel

On Wed, Oct 19, 2016 at 5:= 12 PM, Fahar Abbas <fahar.abbas@enterprisedb.com>= wrote:


On Wed, Oc= t 19, 2016 at 4:03 PM, Fahar Abbas <fahar.abbas@enterprisedb.co= m> wrote:
=

On Wed,= Oct 19, 2016 at 3:55 PM, Ashesh Vashi <ashesh.vashi@enterpris= edb.com> wrote:
Hi Fahar,

Please log= the case on redmine.
Please find the attached patch, ple= ase apply it locally, and test it.

And, please upd= ate the case, and this mail chain accordingly.
This is resolved now and no e= rror message displayed when we apply the patch that is already shared.
=

Sure Will test the patch and update the= status accordingly.
<= div class=3D"m_-8453965415581444024m_3537455806921197709gmail-m_-1620282619= 331950935gmail_signature">
--

Thanks & Regards,

Ashesh Vashi
EnterpriseDB INDIA: Enterprise PostgreSQL Company=

<= br>

<= a href=3D"http://www.linkedin.com/in/asheshvashi" target=3D"_blank">http= ://www.linkedin.com/in/asheshvashi


On Wed, Oct 19, 2016 at 3:47 PM, Fahar A= bbas <fahar.abbas@enterprisedb.com> wrote:
Here i= s the output of if we copy config_local.py and execute python setup.py
pgAdmin 4 - Application Initialisation
=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D


The configuration database - '/home/fahar= /.pgadmin/pgadmin4.db' does not exist.
Entering initial setup m= ode...
NOTE: Configuring authentication for SERVER mode.


=C2= =A0=C2=A0=C2=A0 Enter the email address and password to use for the initial= pgAdmin user=C2=A0=C2=A0=C2=A0=C2=A0 account:

Email address: fahar.abbas@ent= erprisedb.com
Password:
Retype password:
Traceback (most rece= nt call last):
=C2=A0 File "setup.py", line 449, in <module= >
=C2=A0=C2=A0=C2=A0 do_setup(app)
=C2=A0 File "setup.py"= ;, line 96, in do_setup
=C2=A0=C2=A0=C2=A0 password =3D encrypt_password= (p1)
=C2=A0 File "/home/fahar/venv/lib/python3.5/site-packages= /flask_security/utils.py", line 150, in encrypt_password
=C2= =A0=C2=A0=C2=A0 signed =3D get_hmac(password).decode('ascii')<= br>=C2=A0 File "/home/fahar/venv/lib/python3.5/site-packages/flas= k_security/utils.py", line 108, in get_hmac
=C2=A0=C2=A0=C2=A0= 'set to "%s"' % _security.password_hash)
RuntimeError= : The configuration value `SECURITY_PASSWORD_SALT` must not be None when th= e value of `SECURITY_PASSWORD_HASH` is set to "pbkdf2_sha512"
= python setup.py
pgAdmin 4 - Application Initialisation
=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D

User can not do any se= tup for web based now.


The configuration database - &= #39;/home/fahar/.pgadmin/pgadmin4.db' does not exist.
Entering = initial setup mode...
NOTE: Configuring authentication for SERVER mode.<= br>

=C2=A0=C2=A0=C2=A0 Enter the email address and password to use f= or the initial pgAdmin user=C2=A0=C2=A0=C2=A0=C2=A0 account:

Email a= ddress: f= ahar.abbas@enterprisedb.com
Password:
Retype password:
Traceb= ack (most recent call last):
=C2=A0 File "setup.py", line 449,= in <module>
=C2=A0=C2=A0=C2=A0 do_setup(app)
=C2=A0 File "= ;setup.py", line 96, in do_setup
=C2=A0=C2=A0=C2=A0 password =3D en= crypt_password(p1)
=C2=A0 File "/home/fahar/venv/lib/python3.5= /site-packages/flask_security/utils.py", line 150, in encrypt_pas= sword
=C2=A0=C2=A0=C2=A0 signed =3D get_hmac(password).decode('ascii')
=C2=A0 File "/home/fahar/venv/lib/python3.5/site-p= ackages/flask_security/utils.py", line 108, in get_hmac
=C2=A0= =C2=A0=C2=A0 'set to "%s"' % _security.password_hash)
= RuntimeError: The configuration value `SECURITY_PASSWORD_SALT` must not be = None when the value of `SECURITY_PASSWORD_HASH` is set to "pbkdf2_sha5= 12"

On Wed, Oct 19, 2016 at 3:0= 3 PM, Fahar Abbas <fahar.abbas@enterprisedb.com> = wrote:
Dave,

Testing Environment
=C2=A0
=
Ubuntu 16.04 Linux 64:
--------------------------------

pg-AdminIV Development Envir= onment Setu= p for

1)= Install GIT

sudo apt-get install git

2) Install pip3

sudo apt-get install python3-pip

3) Install virtualenv=

sudo pip3 install virtual= env

4) insta= ll below dependency as it is required for psycopg2 & pycrypto module

sudo apt-get i= nstall libpq-dev

sudo apt-get install python3-dev

5) Create virtual environment

<= p dir=3D"ltr" style=3D"line-height:1.38;margin-top:0pt;margin-bottom:0pt"><= font size=3D"2">virtualenv -p python3 venv

6) Create mkdir Projects

7) Clone g= it repo in Projects

git clone http://git.postgresql.org/git/pgadmin4.git=

8) activate virtua= l environment

source venv/bin/activate

9) Install modules

pip3 install -r requirements_py3.txt

10)= Edit the config.py file to config_local.py =C2=A0resides in Project= s\pgA= dmin4\web=C2=A0=C2=A0

1= 1)Now run setup.py file =C2=A0(\Projects\pgAdmin4= \web)

=C2=A0 =C2=A0 python setup.py

If user does = not create config_local.py and do Python setup.py for new Development then = SECURITY_PASSWORD_SALT message is also displayed:

Here is= the output:
-------------------------

python setup.py=
pgAdmin 4 - Application Initialisation
=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D


The configuration database - '/home/fahar= /.pgadmin/pgadmin4.db' does not exist.
Entering initial setup m= ode...
NOTE: Configuring authentication for SERVER mode.


=C2= =A0=C2=A0=C2=A0 Enter the email address and password to use for the initial= pgAdmin user=C2=A0=C2=A0=C2=A0=C2=A0 account:

Email address: fahar.abbas@ent= erprisedb.com
Password:
Retype password:
Traceback (most rece= nt call last):
=C2=A0 File "setup.py", line 449, in <module= >
=C2=A0=C2=A0=C2=A0 do_setup(app)
=C2=A0 File "setup.py"= ;, line 96, in do_setup
=C2=A0=C2=A0=C2=A0 password =3D encrypt_password= (p1)
=C2=A0 File "/home/fahar/venv/lib/python3.5/site-packages= /flask_security/utils.py", line 150, in encrypt_password
=C2= =A0=C2=A0=C2=A0 signed =3D get_hmac(password).decode('ascii')<= br>=C2=A0 File "/home/fahar/venv/lib/python3.5/site-packages/flas= k_security/utils.py", line 108, in get_hmac
=C2=A0=C2=A0=C2=A0= 'set to "%s"' % _security.password_hash)
RuntimeError= : The configuration value `SECURITY_PASSWORD_SALT` must not be None when th= e value of `SECURITY_PASSWORD_HASH` is set to "pbkdf2_sha512"
= (venv) fahar@fahar-virtual-machine:~/Projects/pgadmin4/web$


Is this= expected?

On Wed, Oct 19, 2016 at 1:37 PM= , Fahar Abbas <fahar.abbas@enterprisedb.com> wrot= e:
Sure,

Will test this thoroughly after complete investi= gation.

Kind Regards,

On Wed, Oct 19, 2016 at 1:27 PM, Dave Page = <dpage@pgadmin.or= g> wrote:
Patch applied.

Fahar, can you please = test this thoroughly in desktop and server modes, with both fresh and upgra= ded installations?


Packagers: This change means th= at packages are no longer forced to create a config_local.py file, and ther= e is no longer any need to explicitly set=C2=A0SECURITY_PASSWORD_SALT,=C2=A0S= ECURITY_KEY and=C2=A0CSRF_SESSION_KEY in the config (in fact, they sho= uld be removed for new installations, if you have included them in 1.0)

Thanks.


On Wed, Oct 19, 2016 at 6:46 AM, Ashesh Vas= hi <ashesh.vashi@enterprisedb.com> wrote:
Hi Dav= e,

On Sat, Oct 15, 2016 at 8:02 AM, Dave Page = <dpage@pgadmin.org> wrote:
Hi


On Friday, October 14, 2016, Dave Page <dpage@pgadmin.org> wrote:
Hi

On Thursday, = October 13, 2016, Ashesh Vashi <ashesh.vashi@enterprisedb.com> wrote:
Hi Dave,

On Tue, Oct 11, 2016 at 9:10 PM, Dave Page <= ;dpage@pgadmin.org> wrote:
Hi Ashesh,

Can you please review t= he attached patch, and apply if you're happy with it?
Overall the patch looked good to me.
But - I encounter an i= ssue in 'web' mode, which wont happen with 'runtime'.
=

Steps for reproduction on existing pgAdmin 4 environmen= t with 'web' mode.
- Apply the patch
- Start th= e pgAdmin4 application (stand alone application).
- Open pgAdmin = home page.
- Log out (if already login).

And, you will see an exception.

I have figure out= the issue with the patch.
We were setting the SECURITY_PASSWORD_= SALT, after initializing the Security object.
Hence - it could no= t set the SECURITY_KEY, and SECURITY_PASSWORD_SALT properly.

Hmm.
=C2=A0

I had moved the= Security object initialization after fetching these configurations from th= e database.
I have attached a addon patch for the same.

OK, thanks.
=C2=A0=

Now= - I run into another issue.
Because - the existing password was = hashed using the old SECURITY_PASSWORD_SALT, I am no more able to login to = pgAdmin 4.

I think - we need to think about differ= ent strategy for upgrading the configuration file in the 'web' mode= .
I was thinking - we can store the existing security configurati= ons in the database during upgrade process in 'web' mode.

My concern with that is that= we'll likely be storing the default config values in many cases, thus = for those users, perpetuating the problem.

I guess= what we need to do is re-encrypt the password during the upgrade - however= , that makes me think; we then have both the key and the encrypted password= s in the same database which is clearly not a good idea. Sigh... Needs more= thought.=C2=A0

OK, so I&= #39;ve been thinking about this and experimenting for a couple of hours, as= well as annoying the crap out of Magnus by thinking out loud in his genera= l direction, and it looks like this isn't a major problem as from what = I can see, =C2=A0SECURITY_PASSWORD_SALT is (aside from really being a key n= ot a salt) not the only salting that's done.=C2=A0

=
It looks like it's used system-wide as the key to generate an HMAC= of the users password, which is then passed to passlib which salts and has= hes it. I did some testing, and found that two users with the same password= end up with different hashes in the database, so clearly there is also per= -user salting happening. I also created two users, then dropped the databas= e and created the same user accounts with the same passwords again, and fou= nd that the resulting hashes were different in both databases - thus there = is something else ensuring the hashes are unique across different installat= ions/databases.

So, I believe we can do as you sug= gest and migrate existing values for SECURITY_PASSWORD_SALT, given that the= re's clearly some other per user and per installation/database salting = going on anyway. New installations can have the random value for SECURITY_P= ASSWORD_SALT.
We do not need to generate= the random SECURITY_PASSWORD_SALT during upgrade mode, which was wrong add= ed in my addon patch.

Please find the updated patc= h.

Otherwise - looks good to me.
Please = commit the new patch (if you're ok with the change).


--

Thanks & Regar= ds,

<= span style=3D"font-style:italic">Ashesh Vashi
EnterpriseDB INDIA:=C2=A0= Enterpris= e PostgreSQL Company



I don't believe SECURITY_KE= Y and=C2=A0CSRF_SESSION_KEY are issues either, as they're used for purp= oses that are essentially ephemeral, and thus can be changed during an upgr= ade.

Adding Magnus as I'd appreciate any thoug= hts he may have.

Patch attached - please review (A= shesh, but others too would be appreciated)!

Thank= s.


--
Dave Pa= ge
Blog: http:= //pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterpris= edb.com
The Enterprise PostgreSQL Company





--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @= pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL= Company



--
=
Syed Fahar Abbas
Quali= ty Management Group

EnterpriseDB Corporation
Phone Office: = +92-51-835-8874
Phone Direct: +92-51-8466803
Mobile: +92-333-5409= 707
Skype ID: syed.fahar.abbas
Website: www.enterprisedb.com



--
Syed F= ahar Abbas
Quality Management Group

EnterpriseDB= Corporation
Phone Office: +92-51-835-8874
Phone Direct: +92-51-846680= 3
Mobile: +92-333-5409707
Skype ID: syed.fahar.abbas
Webs= ite: www.enterpri= sedb.com



--
Syed Fahar Abbas
<= div>Quality Management Group

EnterpriseDB Corporation
Phone= Office: +92-51-835-8874
Phone Direct: +92-51-8466803
Mobile: +92= -333-5409707
Skype ID: syed.fahar.abbas
Website: www.enterprisedb.com




--
Syed Fahar Abbas
=
Quality Management Group

EnterpriseDB CorporationPhone Office: +92-51-835-8874
Phone Direct: +92-51-8466803
Mobil= e: +92-333-5409707
Skype ID: syed.fahar.abbas
Website: www.enterprisedb.com=



--
Syed Fahar Abbas
Quality Management Group

EnterpriseDB Corporation
Phone = Office: +92-51-835-8874
Phone Direct: +92-51-8466803
Mobile: +92-333-= 5409707
Skype ID: syed.fahar.abbas
Website: www.enterprisedb.com

--94eb2c03fbc2bd6115053f36b42d--