Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtp (Exim 4.84_2) (envelope-from ) id 1bwqTP-0008Is-0I for pgadmin-hackers@arkaria.postgresql.org; Wed, 19 Oct 2016 12:59:59 +0000 Received: from localhost ([127.0.0.1] helo=postgresql.org) by malur.postgresql.org with smtp (Exim 4.84_2) (envelope-from ) id 1bwqTO-0001SH-DH for pgadmin-hackers@arkaria.postgresql.org; Wed, 19 Oct 2016 12:59:58 +0000 Received: from makus.postgresql.org ([2001:4800:1501:1::229]) by malur.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1bwqQF-0000tq-FF for pgadmin-hackers@postgresql.org; Wed, 19 Oct 2016 12:56:43 +0000 Received: from mail-it0-x22e.google.com ([2607:f8b0:4001:c0b::22e]) by makus.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.84_2) (envelope-from ) id 1bwqQB-00081C-BZ for pgadmin-hackers@postgresql.org; Wed, 19 Oct 2016 12:56:42 +0000 Received: by mail-it0-x22e.google.com with SMTP id 139so103323143itm.1 for ; Wed, 19 Oct 2016 05:56:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pgadmin-org.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=PmGrijGd8riF4o0qMLyFd9eT13YJYDZgjUXV31//ueQ=; b=gfmgyndI28/2wmVGJOQ1rb4C4KE+4+rqfJBuiIhFH4ONFO35X/jwLVNYv1BwUi7qM9 tbRtQEB0EP0uClNIRK0m/HeubrVOwsLAHOzrg1UIMM9T3oIV872dj7um0IGe7jho+7sR FLmqPKxc1r6dD9E32ysmat1+Rk5YXq0lIKw9UsUtBgyNzCxlwoRjp99SjftNAtc0aXic TelnOVqfrsczYmqDc8WYi3dmI5RI2/fvr1tejWnoWaoq5sR5dVSmUuS/4NOHVYfrXNkj TEzQ+bsQz6A2kNRWQ8FBBj2PrIxfIapmS7wEPABkXkxp33/v0OMXZl9B30wIMcau5I1M eg3w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=PmGrijGd8riF4o0qMLyFd9eT13YJYDZgjUXV31//ueQ=; b=arn0HtPMjwXx5H7p8G6d6/nrRnivpC5gMSEc9oYLIye45J0H+3RZTGH9HUembqK9Cj PJDpBZS/9qKqOHviYyIjEGKLrIknAQGZVKP1j8BTUqkdQ3V0N/5OX+hFmi3ZsN2jpDlG VEpNoi2km4vgaGR2Vk+KM7eKQt5bV08TxJVJVqiWZfxERK6yJR0PgJxYAN8zY5Em8DJF vY/ZOZtN5oIRPU632yzwWCxPW53XUg/nvzzcaSmpA3lMN/Y0eFI8QJJON/gh+qZIh0qZ lI//F8K0z8HhF/qfVqJ6mP/z/lyTi2xzmtJIlXunsnBpNWUAaft/HphxNYYnwQ64NCXH tFQQ== X-Gm-Message-State: AA6/9Rkgb8MLTt+XavZrIyy/cZdA7aC0etQpoDLPGG/NeWt/YMWPbX24RMIxqFUCYUPX8MpkYHKlZW7tBXVEMw== X-Received: by 10.36.178.27 with SMTP id u27mr6519024ite.113.1476881798163; Wed, 19 Oct 2016 05:56:38 -0700 (PDT) MIME-Version: 1.0 Received: by 10.64.82.130 with HTTP; Wed, 19 Oct 2016 05:56:36 -0700 (PDT) In-Reply-To: References: From: Dave Page Date: Wed, 19 Oct 2016 13:56:36 +0100 Message-ID: Subject: Re: RM1849: Auto-generating security keys To: Neel Patel Cc: Fahar Abbas , Ashesh Vashi , pgadmin-hackers , Josh Berkus , =?UTF-8?B?RGV2cmltIEfDnE5Ew5xa?= , Magnus Hagander , Sandeep Thakkar , Hamid Quddus Akhtar Content-Type: multipart/alternative; boundary=f403045d9978f48daa053f37569d X-Pg-Spam-Score: -2.6 (--) List-Archive: List-Help: List-ID: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: X-Mailing-List: pgadmin-hackers Precedence: bulk Sender: pgadmin-hackers-owner@postgresql.org --f403045d9978f48daa053f37569d Content-Type: text/plain; charset=UTF-8 I assume that's an existing issue with Python 3.5? That file wasn't changed by this patch. On Wed, Oct 19, 2016 at 1:11 PM, Neel Patel wrote: > Hi, > > Just to update for Python 3. > It gives below error while running "pgAdmin4.py". > > ##### > > Traceback (most recent call last): > File "/usr/lib/python3.4/threading.py", line 920, in _bootstrap_inner > self.run() > File "/usr/lib/python3.4/threading.py", line 868, in run > self._target(*self._args, **self._kwargs) > File "/usr/lib/python3.4/socketserver.py", line 620, in > process_request_thread > self.handle_error(request, client_address) > File "/usr/lib/python3.4/socketserver.py", line 617, in > process_request_thread > self.finish_request(request, client_address) > File "/usr/lib/python3.4/socketserver.py", line 344, in finish_request > self.RequestHandlerClass(request, client_address, self) > File "/usr/lib/python3.4/socketserver.py", line 673, in __init__ > self.handle() > File "/home/neel/workspace/pgAdmin4_3_4/lib/python3.4/ > site-packages/werkzeug/serving.py", line 200, in handle > rv = BaseHTTPRequestHandler.handle(self) > File "/usr/lib/python3.4/http/server.py", line 398, in handle > self.handle_one_request() > File "/home/neel/workspace/pgAdmin4_3_4/lib/python3.4/ > site-packages/werkzeug/serving.py", line 235, in handle_one_request > return self.run_wsgi() > File "/home/neel/workspace/pgAdmin4_3_4/lib/python3.4/ > site-packages/werkzeug/serving.py", line 177, in run_wsgi > execute(self.server.app) > File "/home/neel/workspace/pgAdmin4_3_4/lib/python3.4/ > site-packages/werkzeug/serving.py", line 165, in execute > application_iter = app(environ, start_response) > File "/home/neel/workspace/pgAdmin4_3_4/lib/python3.4/site-packages/flask/app.py", > line 2000, in __call__ > return self.wsgi_app(environ, start_response) > File "/home/neel/workspace/pgAdmin4_3_4/lib/python3.4/site-packages/flask/app.py", > line 1991, in wsgi_app > response = self.make_response(self.handle_exception(e)) > File "/home/neel/workspace/pgAdmin4_3_4/lib/python3.4/site-packages/flask/app.py", > line 1567, in handle_exception > reraise(exc_type, exc_value, tb) > File "/home/neel/workspace/pgAdmin4_3_4/lib/python3.4/ > site-packages/flask/_compat.py", line 33, in reraise > raise value > File "/home/neel/workspace/pgAdmin4_3_4/lib/python3.4/site-packages/flask/app.py", > line 1988, in wsgi_app > response = self.full_dispatch_request() > File "/home/neel/workspace/pgAdmin4_3_4/lib/python3.4/site-packages/flask/app.py", > line 1643, in full_dispatch_request > response = self.process_response(response) > File "/home/neel/workspace/pgAdmin4_3_4/lib/python3.4/site-packages/flask/app.py", > line 1864, in process_response > self.save_session(ctx.session, response) > File "/home/neel/workspace/pgAdmin4_3_4/lib/python3.4/site-packages/flask/app.py", > line 926, in save_session > return self.session_interface.save_session(self, session, response) > File "/home/neel/Projects/pgAdmin4/pgadmin4_patch/pgadmin4/web/pgadmin/utils/session.py", > line 267, in save_session > self.manager.put(session) > File "/home/neel/Projects/pgAdmin4/pgadmin4_patch/pgadmin4/web/pgadmin/utils/session.py", > line 144, in put > self.parent.put(session) > File "/home/neel/Projects/pgAdmin4/pgadmin4_patch/pgadmin4/web/pgadmin/utils/session.py", > line 214, in put > session.sign(self.secret) > File "/home/neel/Projects/pgAdmin4/pgadmin4_patch/pgadmin4/web/pgadmin/utils/session.py", > line 71, in sign > self.hmac_digest = _calc_hmac('%s:%s' % (self.sid, self.randval), > secret) > File "/home/neel/Projects/pgAdmin4/pgadmin4_patch/pgadmin4/web/pgadmin/utils/session.py", > line 44, in _calc_hmac > secret.encode(), body.encode(), hashlib.sha1 > AttributeError: 'bytes' object has no attribute 'encode' > ####### > > Thanks, > Neel Patel > > On Wed, Oct 19, 2016 at 5:12 PM, Fahar Abbas > wrote: > >> >> >> On Wed, Oct 19, 2016 at 4:03 PM, Fahar Abbas < >> fahar.abbas@enterprisedb.com> wrote: >> >>> >>> >>> On Wed, Oct 19, 2016 at 3:55 PM, Ashesh Vashi < >>> ashesh.vashi@enterprisedb.com> wrote: >>> >>>> Hi Fahar, >>>> >>>> Please log the case on redmine. >>>> >>> https://redmine.postgresql.org/issues/1871 >>> >>>> Please find the attached patch, please apply it locally, and test it. >>>> >>>> And, please update the case, and this mail chain accordingly. >>>> >>> This is resolved now and no error message displayed when we apply the >> patch that is already shared. >> >>> >>>> Sure Will test the patch and update the status accordingly. >>> >>>> -- >>>> >>>> Thanks & Regards, >>>> >>>> Ashesh Vashi >>>> EnterpriseDB INDIA: Enterprise PostgreSQL Company >>>> >>>> >>>> >>>> *http://www.linkedin.com/in/asheshvashi* >>>> >>>> >>>> On Wed, Oct 19, 2016 at 3:47 PM, Fahar Abbas < >>>> fahar.abbas@enterprisedb.com> wrote: >>>> >>>>> Here is the output of if we copy config_local.py and execute python >>>>> setup.py >>>>> pgAdmin 4 - Application Initialisation >>>>> ====================================== >>>>> >>>>> >>>>> The configuration database - '/home/fahar/.pgadmin/pgadmin4.db' does >>>>> not exist. >>>>> Entering initial setup mode... >>>>> NOTE: Configuring authentication for SERVER mode. >>>>> >>>>> >>>>> Enter the email address and password to use for the initial >>>>> pgAdmin user account: >>>>> >>>>> Email address: fahar.abbas@enterprisedb.com >>>>> Password: >>>>> Retype password: >>>>> Traceback (most recent call last): >>>>> File "setup.py", line 449, in >>>>> do_setup(app) >>>>> File "setup.py", line 96, in do_setup >>>>> password = encrypt_password(p1) >>>>> File "/home/fahar/venv/lib/python3.5/site-packages/flask_security/utils.py", >>>>> line 150, in encrypt_password >>>>> signed = get_hmac(password).decode('ascii') >>>>> File "/home/fahar/venv/lib/python3.5/site-packages/flask_security/utils.py", >>>>> line 108, in get_hmac >>>>> 'set to "%s"' % _security.password_hash) >>>>> RuntimeError: The configuration value `SECURITY_PASSWORD_SALT` must >>>>> not be None when the value of `SECURITY_PASSWORD_HASH` is set to >>>>> "pbkdf2_sha512" >>>>> python setup.py >>>>> pgAdmin 4 - Application Initialisation >>>>> ====================================== >>>>> >>>>> User can not do any setup for web based now. >>>>> >>>>> >>>>> The configuration database - '/home/fahar/.pgadmin/pgadmin4.db' does >>>>> not exist. >>>>> Entering initial setup mode... >>>>> NOTE: Configuring authentication for SERVER mode. >>>>> >>>>> >>>>> Enter the email address and password to use for the initial >>>>> pgAdmin user account: >>>>> >>>>> Email address: fahar.abbas@enterprisedb.com >>>>> Password: >>>>> Retype password: >>>>> Traceback (most recent call last): >>>>> File "setup.py", line 449, in >>>>> do_setup(app) >>>>> File "setup.py", line 96, in do_setup >>>>> password = encrypt_password(p1) >>>>> File "/home/fahar/venv/lib/python3.5/site-packages/flask_security/utils.py", >>>>> line 150, in encrypt_password >>>>> signed = get_hmac(password).decode('ascii') >>>>> File "/home/fahar/venv/lib/python3.5/site-packages/flask_security/utils.py", >>>>> line 108, in get_hmac >>>>> 'set to "%s"' % _security.password_hash) >>>>> RuntimeError: The configuration value `SECURITY_PASSWORD_SALT` must >>>>> not be None when the value of `SECURITY_PASSWORD_HASH` is set to >>>>> "pbkdf2_sha512" >>>>> >>>>> On Wed, Oct 19, 2016 at 3:03 PM, Fahar Abbas < >>>>> fahar.abbas@enterprisedb.com> wrote: >>>>> >>>>>> Dave, >>>>>> >>>>>> Testing Environment >>>>>> >>>>>> Ubuntu 16.04 Linux 64: >>>>>> -------------------------------- >>>>>> >>>>>> pg-AdminIV Development Environment Setup for Ubuntu : >>>>>> >>>>>> >>>>>> 1) Install GIT >>>>>> >>>>>> sudo apt-get install git >>>>>> >>>>>> 2) Install pip3 >>>>>> >>>>>> sudo apt-get install python3-pip >>>>>> >>>>>> 3) Install virtualenv >>>>>> >>>>>> sudo pip3 install virtualenv >>>>>> >>>>>> 4) install below dependency as it is required for psycopg2 & pycrypto >>>>>> module >>>>>> >>>>>> sudo apt-get install libpq-dev >>>>>> >>>>>> sudo apt-get install python3-dev >>>>>> >>>>>> 5) Create virtual environment >>>>>> >>>>>> virtualenv -p python3 venv >>>>>> >>>>>> 6) Create mkdir Projects >>>>>> >>>>>> 7) Clone git repo in Projects >>>>>> >>>>>> git clone http://git.postgresql.org/git/pgadmin4.git >>>>>> >>>>>> 8) activate virtual environment >>>>>> >>>>>> source venv/bin/activate >>>>>> >>>>>> 9) Install modules >>>>>> >>>>>> pip3 install -r requirements_py3.txt >>>>>> >>>>>> *10) Edit the config.py file to config_local.py resides in >>>>>> Projects\pgAdmin4\web * >>>>>> >>>>>> 11)Now run setup.py file (\Projects\pgAdmin4\web) >>>>>> python setup.py >>>>>> >>>>>> If user does not create config_local.py and do Python setup.py for >>>>>> new Development then SECURITY_PASSWORD_SALT message is also displayed: >>>>>> >>>>>> Here is the output: >>>>>> ------------------------- >>>>>> >>>>>> python setup.py >>>>>> pgAdmin 4 - Application Initialisation >>>>>> ====================================== >>>>>> >>>>>> >>>>>> The configuration database - '/home/fahar/.pgadmin/pgadmin4.db' does >>>>>> not exist. >>>>>> Entering initial setup mode... >>>>>> NOTE: Configuring authentication for SERVER mode. >>>>>> >>>>>> >>>>>> Enter the email address and password to use for the initial >>>>>> pgAdmin user account: >>>>>> >>>>>> Email address: fahar.abbas@enterprisedb.com >>>>>> Password: >>>>>> Retype password: >>>>>> Traceback (most recent call last): >>>>>> File "setup.py", line 449, in >>>>>> do_setup(app) >>>>>> File "setup.py", line 96, in do_setup >>>>>> password = encrypt_password(p1) >>>>>> File "/home/fahar/venv/lib/python3.5/site-packages/flask_security/utils.py", >>>>>> line 150, in encrypt_password >>>>>> signed = get_hmac(password).decode('ascii') >>>>>> File "/home/fahar/venv/lib/python3.5/site-packages/flask_security/utils.py", >>>>>> line 108, in get_hmac >>>>>> 'set to "%s"' % _security.password_hash) >>>>>> RuntimeError: The configuration value `SECURITY_PASSWORD_SALT` must >>>>>> not be None when the value of `SECURITY_PASSWORD_HASH` is set to >>>>>> "pbkdf2_sha512" >>>>>> (venv) fahar@fahar-virtual-machine:~/Projects/pgadmin4/web$ >>>>>> >>>>>> >>>>>> Is this expected? >>>>>> >>>>>> On Wed, Oct 19, 2016 at 1:37 PM, Fahar Abbas < >>>>>> fahar.abbas@enterprisedb.com> wrote: >>>>>> >>>>>>> Sure, >>>>>>> >>>>>>> Will test this thoroughly after complete investigation. >>>>>>> >>>>>>> Kind Regards, >>>>>>> >>>>>>> On Wed, Oct 19, 2016 at 1:27 PM, Dave Page >>>>>>> wrote: >>>>>>> >>>>>>>> Patch applied. >>>>>>>> >>>>>>>> Fahar, can you please test this thoroughly in desktop and server >>>>>>>> modes, with both fresh and upgraded installations? >>>>>>>> >>>>>>>> https://redmine.postgresql.org/issues/1849 >>>>>>>> >>>>>>>> Packagers: This change means that packages are no longer forced to >>>>>>>> create a config_local.py file, and there is no longer any need to >>>>>>>> explicitly set SECURITY_PASSWORD_SALT, SECURITY_KEY >>>>>>>> and CSRF_SESSION_KEY in the config (in fact, they should be removed for new >>>>>>>> installations, if you have included them in 1.0) >>>>>>>> >>>>>>>> Thanks. >>>>>>>> >>>>>>>> >>>>>>>> On Wed, Oct 19, 2016 at 6:46 AM, Ashesh Vashi < >>>>>>>> ashesh.vashi@enterprisedb.com> wrote: >>>>>>>> >>>>>>>>> Hi Dave, >>>>>>>>> >>>>>>>>> On Sat, Oct 15, 2016 at 8:02 AM, Dave Page >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Hi >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Friday, October 14, 2016, Dave Page wrote: >>>>>>>>>> >>>>>>>>>>> Hi >>>>>>>>>>> >>>>>>>>>>> On Thursday, October 13, 2016, Ashesh Vashi < >>>>>>>>>>> ashesh.vashi@enterprisedb.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi Dave, >>>>>>>>>>>> >>>>>>>>>>>> On Tue, Oct 11, 2016 at 9:10 PM, Dave Page >>>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hi Ashesh, >>>>>>>>>>>>> >>>>>>>>>>>>> Can you please review the attached patch, and apply if you're >>>>>>>>>>>>> happy with it? >>>>>>>>>>>>> >>>>>>>>>>>> Overall the patch looked good to me. >>>>>>>>>>>> But - I encounter an issue in 'web' mode, which wont happen >>>>>>>>>>>> with 'runtime'. >>>>>>>>>>>> >>>>>>>>>>>> Steps for reproduction on existing pgAdmin 4 environment with >>>>>>>>>>>> 'web' mode. >>>>>>>>>>>> - Apply the patch >>>>>>>>>>>> - Start the pgAdmin4 application (stand alone application). >>>>>>>>>>>> - Open pgAdmin home page. >>>>>>>>>>>> - Log out (if already login). >>>>>>>>>>>> >>>>>>>>>>>> And, you will see an exception. >>>>>>>>>>>> >>>>>>>>>>>> I have figure out the issue with the patch. >>>>>>>>>>>> We were setting the SECURITY_PASSWORD_SALT, after initializing >>>>>>>>>>>> the Security object. >>>>>>>>>>>> Hence - it could not set the SECURITY_KEY, and >>>>>>>>>>>> SECURITY_PASSWORD_SALT properly. >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Hmm. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> I had moved the Security object initialization after fetching >>>>>>>>>>>> these configurations from the database. >>>>>>>>>>>> I have attached a addon patch for the same. >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> OK, thanks. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Now - I run into another issue. >>>>>>>>>>>> Because - the existing password was hashed using the old >>>>>>>>>>>> SECURITY_PASSWORD_SALT, I am no more able to login to pgAdmin 4. >>>>>>>>>>>> >>>>>>>>>>>> I think - we need to think about different strategy for >>>>>>>>>>>> upgrading the configuration file in the 'web' mode. >>>>>>>>>>>> I was thinking - we can store the existing security >>>>>>>>>>>> configurations in the database during upgrade process in 'web' mode. >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> My concern with that is that we'll likely be storing the default >>>>>>>>>>> config values in many cases, thus for those users, perpetuating the problem. >>>>>>>>>>> >>>>>>>>>>> I guess what we need to do is re-encrypt the password during the >>>>>>>>>>> upgrade - however, that makes me think; we then have both the key and the >>>>>>>>>>> encrypted passwords in the same database which is clearly not a good idea. >>>>>>>>>>> Sigh... Needs more thought. >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> OK, so I've been thinking about this and experimenting for a >>>>>>>>>> couple of hours, as well as annoying the crap out of Magnus by thinking out >>>>>>>>>> loud in his general direction, and it looks like this isn't a major problem >>>>>>>>>> as from what I can see, SECURITY_PASSWORD_SALT is (aside from really being >>>>>>>>>> a key not a salt) not the only salting that's done. >>>>>>>>>> >>>>>>>>>> It looks like it's used system-wide as the key to generate an >>>>>>>>>> HMAC of the users password, which is then passed to passlib which salts and >>>>>>>>>> hashes it. I did some testing, and found that two users with the same >>>>>>>>>> password end up with different hashes in the database, so clearly there is >>>>>>>>>> also per-user salting happening. I also created two users, then dropped the >>>>>>>>>> database and created the same user accounts with the same passwords again, >>>>>>>>>> and found that the resulting hashes were different in both databases - thus >>>>>>>>>> there is something else ensuring the hashes are unique across different >>>>>>>>>> installations/databases. >>>>>>>>>> >>>>>>>>>> So, I believe we can do as you suggest and migrate existing >>>>>>>>>> values for SECURITY_PASSWORD_SALT, given that there's clearly some other >>>>>>>>>> per user and per installation/database salting going on anyway. New >>>>>>>>>> installations can have the random value for SECURITY_PASSWORD_SALT. >>>>>>>>>> >>>>>>>>> We do not need to generate the random SECURITY_PASSWORD_SALT >>>>>>>>> during upgrade mode, which was wrong added in my addon patch. >>>>>>>>> >>>>>>>>> Please find the updated patch. >>>>>>>>> >>>>>>>>> Otherwise - looks good to me. >>>>>>>>> Please commit the new patch (if you're ok with the change). >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> >>>>>>>>> Thanks & Regards, >>>>>>>>> >>>>>>>>> Ashesh Vashi >>>>>>>>> EnterpriseDB INDIA: Enterprise PostgreSQL Company >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> *http://www.linkedin.com/in/asheshvashi* >>>>>>>>> >>>>>>>>> >>>>>>>>>> >>>>>>>>>> I don't believe SECURITY_KEY and CSRF_SESSION_KEY are issues >>>>>>>>>> either, as they're used for purposes that are essentially ephemeral, and >>>>>>>>>> thus can be changed during an upgrade. >>>>>>>>>> >>>>>>>>>> Adding Magnus as I'd appreciate any thoughts he may have. >>>>>>>>>> >>>>>>>>>> Patch attached - please review (Ashesh, but others too would be >>>>>>>>>> appreciated)! >>>>>>>>>> >>>>>>>>>> Thanks. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Dave Page >>>>>>>>>> Blog: http://pgsnake.blogspot.com >>>>>>>>>> Twitter: @pgsnake >>>>>>>>>> >>>>>>>>>> EnterpriseDB UK: http://www.enterprisedb.com >>>>>>>>>> The Enterprise PostgreSQL Company >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Dave Page >>>>>>>> Blog: http://pgsnake.blogspot.com >>>>>>>> Twitter: @pgsnake >>>>>>>> >>>>>>>> EnterpriseDB UK: http://www.enterprisedb.com >>>>>>>> The Enterprise PostgreSQL Company >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Syed Fahar Abbas >>>>>>> Quality Management Group >>>>>>> >>>>>>> EnterpriseDB Corporation >>>>>>> Phone Office: +92-51-835-8874 >>>>>>> Phone Direct: +92-51-8466803 >>>>>>> Mobile: +92-333-5409707 >>>>>>> Skype ID: syed.fahar.abbas >>>>>>> Website: www.enterprisedb.com >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Syed Fahar Abbas >>>>>> Quality Management Group >>>>>> >>>>>> EnterpriseDB Corporation >>>>>> Phone Office: +92-51-835-8874 >>>>>> Phone Direct: +92-51-8466803 >>>>>> Mobile: +92-333-5409707 >>>>>> Skype ID: syed.fahar.abbas >>>>>> Website: www.enterprisedb.com >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Syed Fahar Abbas >>>>> Quality Management Group >>>>> >>>>> EnterpriseDB Corporation >>>>> Phone Office: +92-51-835-8874 >>>>> Phone Direct: +92-51-8466803 >>>>> Mobile: +92-333-5409707 >>>>> Skype ID: syed.fahar.abbas >>>>> Website: www.enterprisedb.com >>>>> >>>> >>>> >>> >>> >>> -- >>> Syed Fahar Abbas >>> Quality Management Group >>> >>> EnterpriseDB Corporation >>> Phone Office: +92-51-835-8874 >>> Phone Direct: +92-51-8466803 >>> Mobile: +92-333-5409707 >>> Skype ID: syed.fahar.abbas >>> Website: www.enterprisedb.com >>> >> >> >> >> -- >> Syed Fahar Abbas >> Quality Management Group >> >> EnterpriseDB Corporation >> Phone Office: +92-51-835-8874 >> Phone Direct: +92-51-8466803 >> Mobile: +92-333-5409707 >> Skype ID: syed.fahar.abbas >> Website: www.enterprisedb.com >> > > -- Dave Page Blog: http://pgsnake.blogspot.com Twitter: @pgsnake EnterpriseDB UK: http://www.enterprisedb.com The Enterprise PostgreSQL Company --f403045d9978f48daa053f37569d Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
I assume that's an existing issue with Python 3.5? Tha= t file wasn't changed by this patch.
On Wed, Oct 19, 2016 at 1:11 PM, Neel Patel <neel.patel@enterprisedb.com> wrote:
Hi,

Just to update= for Python 3.
It gives below error while running "pgAdmin4.= py".

#####

Traceback (most recent call last):
=C2=A0 Fil= e "/usr/lib/python3.4/threading.py", line 920, in _bootstrap= _inner
=C2=A0 =C2=A0 self.run()
=C2=A0 File "/usr/= lib/python3.4/threading.py", line 868, in run
=C2=A0 = =C2=A0 self._target(*self._args, **self._kwargs)
=C2=A0 File &quo= t;/usr/lib/python3.4/socketserver.py", line 620, in process_reque= st_thread
=C2=A0 =C2=A0 self.handle_error(request, client_address= )
=C2=A0 File "/usr/lib/python3.4/socketserver.py"= , line 617, in process_request_thread
=C2=A0 =C2=A0 self.finish_r= equest(request, client_address)
=C2=A0 File "/usr/lib/python= 3.4/socketserver.py", line 344, in finish_request
=C2= =A0 =C2=A0 self.RequestHandlerClass(request, client_address, self)
=C2=A0 File "/usr/lib/python3.4/socketserver.py", lin= e 673, in __init__
=C2=A0 =C2=A0 self.handle()
=C2=A0 F= ile "/home/neel/workspace/pgAdmin4_3_4/lib/python3.4/site-pa= ckages/werkzeug/serving.py", line 200, in handle
=C2=A0= =C2=A0 rv =3D BaseHTTPRequestHandler.handle(self)
=C2=A0 Fi= le "/usr/lib/python3.4/http/server.py", line 398, in handle<= /div>
=C2=A0 =C2=A0 self.handle_one_request()
=C2=A0 File &qu= ot;/home/neel/workspace/pgAdmin4_3_4/lib/python3.4/site-packages/= werkzeug/serving.py", line 235, in handle_one_request
= =C2=A0 =C2=A0 return self.run_wsgi()
=C2=A0 File "/home/neel= /workspace/pgAdmin4_3_4/lib/python3.4/site-packages/werkzeug/serving.py", line 177, in run_wsgi
=C2=A0 =C2=A0 execute(se= lf.server.app)
=C2=A0 File "/home/neel/workspace/pgAdmi= n4_3_4/lib/python3.4/site-packages/werkzeug/serving.py", lin= e 165, in execute
=C2=A0 =C2=A0 application_iter =3D app(environ,= start_response)
=C2=A0 File "/home/neel/workspace/pgAd= min4_3_4/lib/python3.4/site-packages/flask/app.py", line 2000, in= __call__
=C2=A0 =C2=A0 return self.wsgi_app(environ, start_respo= nse)
=C2=A0 File "/home/neel/workspace/pgAdmin4_3_4/lib= /python3.4/site-packages/flask/app.py", line 1991, in wsgi_app
=C2=A0 =C2=A0 response =3D self.make_response(self.handle_exce= ption(e))
=C2=A0 File "/home/neel/workspace/pgAdmin4_3_= 4/lib/python3.4/site-packages/flask/app.py", line 1567, in handle= _exception
=C2=A0 =C2=A0 reraise(exc_type, exc_value, tb)
=C2=A0 File "/home/neel/workspace/pgAdmin4_3_4/lib/python3.4/<= wbr>site-packages/flask/_compat.py", line 33, in reraise
=C2=A0 =C2=A0 raise value
=C2=A0 File "/home/neel/workspac= e/pgAdmin4_3_4/lib/python3.4/site-packages/flask/app.py", li= ne 1988, in wsgi_app
=C2=A0 =C2=A0 response =3D self.full_dispatc= h_request()
=C2=A0 File "/home/neel/workspace/pgAdmin4_= 3_4/lib/python3.4/site-packages/flask/app.py", line 1643, in full= _dispatch_request
=C2=A0 =C2=A0 response =3D self.process_respons= e(response)
=C2=A0 File "/home/neel/workspace/pgAd= min4_3_4/lib/python3.4/site-packages/flask/app.py", line 1864, in= process_response
=C2=A0 =C2=A0 self.save_session(ctx.session, re= sponse)
=C2=A0 File "/home/neel/workspace/pgAdmin4_3_4/= lib/python3.4/site-packages/flask/app.py", line 926, in save_sess= ion
=C2=A0 =C2=A0 return self.session_interface.save_session= (self, session, response)
=C2=A0 File "/home/neel/Projects/p= gAdmin4/pgadmin4_patch/pgadmin4/web/pgadmin/utils/session.py"= ;, line 267, in save_session
=C2=A0 =C2=A0 self.manager.put(sessi= on)
=C2=A0 File "/home/neel/Projects/pgAdmin4/pgadmin4_= patch/pgadmin4/web/pgadmin/utils/session.py", line 144, in put
=C2=A0 =C2=A0 self.parent.put(session)
=C2=A0 File "= /home/neel/Projects/pgAdmin4/pgadmin4_patch/pgadmin4/web/pgadmin/= utils/session.py", line 214, in put
=C2=A0 =C2=A0 session.si= gn(self.secret)
=C2=A0 File "/home/neel/Projects/pgAdmin4/pgadmin4_patch/pgadmin4/web/pgadmin/utils/session.py", line 71= , in sign
=C2=A0 =C2=A0 self.hmac_digest =3D _calc_hmac('%s:%= s' % (self.sid, self.randval), secret)
=C2=A0 File "/hom= e/neel/Projects/pgAdmin4/pgadmin4_patch/pgadmin4/web/pgadmin/util= s/session.py", line 44, in _calc_hmac
=C2=A0 =C2=A0 secret.e= ncode(), body.encode(), hashlib.sha1
AttributeError: 'bytes&#= 39; object has no attribute 'encode'
=C2=A0#######<= /div>

Thanks,
Neel Patel

On Wed, Oct 19, 2016 at 5:12 PM, Fahar Abbas <faha= r.abbas@enterprisedb.com> wrote:


On Wed, Oct 19, 2016 at 4:03 PM, Fahar Abbas <f= ahar.abbas@enterprisedb.com> wrote:


On Wed, Oct 19, 2016 at 3:55 PM, Ashesh Vashi <ashesh.vashi@enterprisedb.com> wrote:
Hi Fahar,

Please log the case on redmine.
<= div>https://redmine.postgresql.org/issues/1871
Please f= ind the attached patch, please apply it locally, and test it.
And, please update the case, and this mail chain accordingly.
Th= is is resolved now and no error message displayed when we apply the patch t= hat is already shared.

=
Sure Will test the patch and update the stat= us accordingly.

--

= Thanks & Regards,

As= hesh Vashi
EnterpriseDB INDIA: Enterprise PostgreSQL Company

<= br>

<= a href=3D"http://www.linkedin.com/in/asheshvashi" target=3D"_blank">http= ://www.linkedin.com/in/asheshvashi


On Wed, Oct 19, 201= 6 at 3:47 PM, Fahar Abbas <fahar.abbas@enterprisedb.com>= wrote:
Here is the output of if we copy config_local.py and execute= python setup.py
pgAdmin 4 - Application Initialisation
=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D


The configuration databas= e - '/home/fahar/.pgadmin/pgadmin4.db' does not exist.
Ente= ring initial setup mode...
NOTE: Configuring authentication for SERVER m= ode.


=C2=A0=C2=A0=C2=A0 Enter the email address and password to = use for the initial pgAdmin user=C2=A0=C2=A0=C2=A0=C2=A0 account:

Em= ail address: fahar.abbas@enterprisedb.com
Password:
Retype password:
T= raceback (most recent call last):
=C2=A0 File "setup.py", line= 449, in <module>
=C2=A0=C2=A0=C2=A0 do_setup(app)
=C2=A0 File = "setup.py", line 96, in do_setup
=C2=A0=C2=A0=C2=A0 password = =3D encrypt_password(p1)
=C2=A0 File "/home/fahar/venv/lib/python3.= 5/site-packages/flask_security/utils.py", line 150, in encry= pt_password
=C2=A0=C2=A0=C2=A0 signed =3D get_hmac(password).decode('= ;ascii')
=C2=A0 File "/home/fahar/venv/lib/python3.5/= site-packages/flask_security/utils.py", line 108, in get_hmac
= =C2=A0=C2=A0=C2=A0 'set to "%s"' % _security.password_has= h)
RuntimeError: The configuration value `SECURITY_PASSWORD_SALT` must n= ot be None when the value of `SECURITY_PASSWORD_HASH` is set to "pbkdf= 2_sha512"
python setup.py
pgAdmin 4 - Application Initialisation=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D

User= can not do any setup for web based now.


The configur= ation database - '/home/fahar/.pgadmin/pgadmin4.db' does not e= xist.
Entering initial setup mode...
NOTE: Configuring authentication= for SERVER mode.


=C2=A0=C2=A0=C2=A0 Enter the email address and= password to use for the initial pgAdmin user=C2=A0=C2=A0=C2=A0=C2=A0 accou= nt:

Email address: fahar.abbas@enterprisedb.com
Password:
Retype p= assword:
Traceback (most recent call last):
=C2=A0 File "setup.p= y", line 449, in <module>
=C2=A0=C2=A0=C2=A0 do_setup(app)=C2=A0 File "setup.py", line 96, in do_setup
=C2=A0=C2=A0=C2= =A0 password =3D encrypt_password(p1)
=C2=A0 File "/home/fahar/venv= /lib/python3.5/site-packages/flask_security/utils.py", line = 150, in encrypt_password
=C2=A0=C2=A0=C2=A0 signed =3D get_hmac(password= ).decode('ascii')
=C2=A0 File "/home/fahar/venv/lib/py= thon3.5/site-packages/flask_security/utils.py", line 108, in= get_hmac
=C2=A0=C2=A0=C2=A0 'set to "%s"' % _security= .password_hash)
RuntimeError: The configuration value `SECURITY_PASSWORD= _SALT` must not be None when the value of `SECURITY_PASSWORD_HASH` is set t= o "pbkdf2_sha512"

On Wed, Oct 19, 2016 at 3:03 PM, Fahar Abba= s <fahar.abbas@enterprisedb.com> wrote:
Dave= ,

Testing Environment
=C2=A0
Ubuntu 16.= 04 Linux 64:
--------------------------------

pg-AdminIV Development Environment Setup for Ubuntu=C2=A0 :


1) Install GIT

= sudo apt-get install git

2) Install pip3

=

= sudo apt-get install python3-= pip

3) Insta= ll virtualenv

sudo pip3 install virtualenv

4) install below dependency as it is required for psycop= g2 & pycrypto module

sudo apt-get install libpq-dev

sudo apt-get install python3-dev

5) Create virtual env= ironment

vir= tualenv -p python3 venv

= 6) Create mkdir Projects

7) Clone git repo in Projects

git clone http://git.postgresql.org= /git/pgadmin4.git

8) activate virtual environment

source venv/bin/activat= e

9) Install= modules

pip= 3 install -r requirements_py3.txt

10) Edit the config.p= y file to config_local.py =C2=A0resides in Projects\pgAdmin= 4\web=C2=A0=C2=A0

11)= Now run setup.py file =C2=A0(\Projects= \pgAdmin4\web)

=C2=A0 =C2=A0 python <= span>setup.py

If user does not create conf= ig_local.py and do Python setup.py for new Development then SECURITY_PASSWO= RD_SALT message is also displayed:

Here is the output:-------------------------

python setup.py
pgAdmin 4 = - Application Initialisation
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D


The configuration database - '/home/fahar/.pgadmin/pgadm= in4.db' does not exist.
Entering initial setup mode...
NOTE:= Configuring authentication for SERVER mode.


=C2=A0=C2=A0=C2=A0 = Enter the email address and password to use for the initial pgAdmin user=C2= =A0=C2=A0=C2=A0=C2=A0 account:

Email address: fahar.abbas@enterprisedb.com
Password:
Retype password:
Traceback (most recent call last):=C2=A0 File "setup.py", line 449, in <module>
=C2=A0= =C2=A0=C2=A0 do_setup(app)
=C2=A0 File "setup.py", line 96, in= do_setup
=C2=A0=C2=A0=C2=A0 password =3D encrypt_password(p1)
=C2=A0= File "/home/fahar/venv/lib/python3.5/site-packages/flask_securit= y/utils.py", line 150, in encrypt_password
=C2=A0=C2=A0=C2=A0 = signed =3D get_hmac(password).decode('ascii')
=C2=A0 File &= quot;/home/fahar/venv/lib/python3.5/site-packages/flask_security/= utils.py", line 108, in get_hmac
=C2=A0=C2=A0=C2=A0 'set to &qu= ot;%s"' % _security.password_hash)
RuntimeError: The configurat= ion value `SECURITY_PASSWORD_SALT` must not be None when the value of `SECU= RITY_PASSWORD_HASH` is set to "pbkdf2_sha512"
(venv) fahar@fah= ar-virtual-machine:~/Projects/pgadmin4/web$


Is this expected?
=
On Wed, Oct 19, 2016 at = 1:37 PM, Fahar Abbas <fahar.abbas@enterprisedb.com> wrote:
Sure,

Will test this thoroughly after complete = investigation.

Kind Regards,

On Wed, Oct 19, 2016 at 1:27 PM, = Dave Page <dpage@pgadmin.org> wrote:
Patch applied.

Fahar, can you please test this thoroughly in desktop and server mode= s, with both fresh and upgraded installations?


Pac= kagers: This change means that packages are no longer forced to create a co= nfig_local.py file, and there is no longer any need to explicitly set=C2=A0= SECURITY_PASSWORD_SALT,=C2=A0SECURITY_KEY and=C2=A0CSRF_SESSION_KEY in t= he config (in fact, they should be removed for new installations, if you ha= ve included them in 1.0)

Thanks.


On Wed, Oct 19, 2016 at 6:46 AM, Ashesh Vashi <<= a href=3D"mailto:ashesh.vashi@enterprisedb.com" target=3D"_blank">ashesh.va= shi@enterprisedb.com> wrote:
Hi Dave,

On Sat, Oct 15, 2016 at 8:02 AM, Dave Page <dpa= ge@pgadmin.org> wrote:
Hi


On Friday, October 14, 2016, Dave Page &l= t;dpage@pgadmin.org<= /a>> wrote:
Hi
<= br>On Thursday, October 13, 2016, Ashesh Vashi <ashesh.vashi@enterpri= sedb.com> wrote:
Hi Dave,

<= div class=3D"gmail_quote">On Tue, Oct 11, 2016 at 9:10 PM, Dave Page <dpage@pgadmin.org> wrote:
Hi Ashesh,

Can you = please review the attached patch, and apply if you're happy with it?
Overall the patch looked good to me.
But - I= encounter an issue in 'web' mode, which wont happen with 'runt= ime'.

Steps for reproduction on existing pgAdm= in 4 environment with 'web' mode.
- Apply the patch
=
- Start the pgAdmin4 application (stand alone application).
= - Open pgAdmin home page.
- Log out (if already login).

And, you will see an exception.

I = have figure out the issue with the patch.
We were setting the SEC= URITY_PASSWORD_SALT, after initializing the Security object.
Henc= e - it could not set the SECURITY_KEY, and SECURITY_PASSWORD_SALT properly.=

Hmm.
=C2= =A0

= I had moved the Security object initialization after fetching these configu= rations from the database.
I have attached a addon patch for the = same.

OK, thanks.
=C2=A0
Now - I run into another issue.
Because - the existin= g password was hashed using the old SECURITY_PASSWORD_SALT, I am no more ab= le to login to pgAdmin 4.

I think - we need to thi= nk about different strategy for upgrading the configuration file in the = 9;web' mode.
I was thinking - we can store the existing secur= ity configurations in the database during upgrade process in 'web' = mode.

My concern wi= th that is that we'll likely be storing the default config values in ma= ny cases, thus for those users, perpetuating the problem.

I guess what we need to do is re-encrypt the password during the up= grade - however, that makes me think; we then have both the key and the enc= rypted passwords in the same database which is clearly not a good idea. Sig= h... Needs more thought.=C2=A0

OK, so I've been thinking about this and experimenting for a coup= le of hours, as well as annoying the crap out of Magnus by thinking out lou= d in his general direction, and it looks like this isn't a major proble= m as from what I can see, =C2=A0SECURITY_PASSWORD_SALT is (aside from reall= y being a key not a salt) not the only salting that's done.=C2=A0
=

It looks like it's used system-wide as the key to g= enerate an HMAC of the users password, which is then passed to passlib whic= h salts and hashes it. I did some testing, and found that two users with th= e same password end up with different hashes in the database, so clearly th= ere is also per-user salting happening. I also created two users, then drop= ped the database and created the same user accounts with the same passwords= again, and found that the resulting hashes were different in both database= s - thus there is something else ensuring the hashes are unique across diff= erent installations/databases.

So, I believe we ca= n do as you suggest and migrate existing values for SECURITY_PASSWORD_SALT,= given that there's clearly some other per user and per installation/da= tabase salting going on anyway. New installations can have the random value= for SECURITY_PASSWORD_SALT.
We do not n= eed to generate the random SECURITY_PASSWORD_SALT during upgrade mode, whic= h was wrong added in my addon patch.

Please find t= he updated patch.




--
Dave Page
Blog: http://pgsnake.blogspot.= com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The = Enterprise PostgreSQL Company



--
Syed Fahar Abbas
Quality Management Group

= EnterpriseDB Corporation
Phone Office: +92-51-835-8874
Phone Direct: = += 92-51-8466803
Mobile: +92-333-5409707
Skype ID: syed.fahar.a= bbas
Website: = www.enterprisedb.com



--
Syed Fahar Abbas
Quality Management Group
<= br>
EnterpriseDB Corporation
Phone Office: +92-51-835-8874
Phone= Direct: +92-51-8466803
Mobile: +92-333-5409707
Skype ID: sye= d.fahar.abbas
Website: www.enterprisedb.com



--
Syed F= ahar Abbas
Quality Management Group

EnterpriseDB= Corporation
Phone Office: +92-51-835-8874
Phone Direct: +92-51-846680= 3
Mobile: +92-333-5409707
Skype ID: syed.fahar.abbas
Webs= ite: www.enterpri= sedb.com




--
=
Syed Fahar Abbas
Quality Management Gro= up

EnterpriseDB Corporation
Phone Office: +92-51-835-8874=
Phone Direct: +92-51-8466803
Mobile: +92-333-5409707
Skyp= e ID: syed.fahar.abbas
Website: www.enterprisedb.com
=


--
Syed Fahar Abbas
Quality Management Group
=
EnterpriseDB Corporation
Phone Office: +92-51-835-8874
Phone = Direct: +92-51-8466803
Mobile: +92-333-5409707
Skype ID: syed= .fahar.abbas
Website: www.enterprisedb.com




--
=
Dave Page=
Blog: http://= pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprised= b.com
The Enterprise PostgreSQL Company
--f403045d9978f48daa053f37569d--