Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtp (Exim 4.84_2) (envelope-from ) id 1bwnif-0005dW-Qm for pgadmin-hackers@arkaria.postgresql.org; Wed, 19 Oct 2016 10:03:34 +0000 Received: from localhost ([127.0.0.1] helo=postgresql.org) by malur.postgresql.org with smtp (Exim 4.84_2) (envelope-from ) id 1bwnif-0007tD-1M for pgadmin-hackers@arkaria.postgresql.org; Wed, 19 Oct 2016 10:03:33 +0000 Received: from makus.postgresql.org ([2001:4800:1501:1::229]) by malur.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1bwnid-0007t6-Vf for pgadmin-hackers@postgresql.org; Wed, 19 Oct 2016 10:03:32 +0000 Received: from mail-lf0-x229.google.com ([2a00:1450:4010:c07::229]) by makus.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.84_2) (envelope-from ) id 1bwniV-0004ik-Ts for pgadmin-hackers@postgresql.org; Wed, 19 Oct 2016 10:03:30 +0000 Received: by mail-lf0-x229.google.com with SMTP id b81so16366240lfe.1 for ; Wed, 19 Oct 2016 03:03:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=enterprisedb-com.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=NsOkT0Cuq1Y0XzPa9XKC2sgmpa2tm9deyAWtcg8gRZ8=; b=c3vH7KsxXFEINavO1zuoDUr+5jytbyWYefloYqE614+1dsRXbW6I5L5jrHPdZCNocZ wlodDhmsuIPeKkRnLcTts11JDxQgj7lKL2kQNTrpWY0qC60QFK2gD2LitJndgBOMYpf6 tRHIOq68ArqtCyhHM6aI7m2BbaSeXDA2ZK5YnWoWxGkWmsBz3LMZKTIZ9wWsPXv8K/SF R1b21OgIAJv3A2JWvYI5M2+KQuu3Ihc1kJLOoN6MU9sDfXYSlxqO6LlssHsOGJOZ281U 6iCxSInnmavPdo1YLvn0j+6th90ufafaM8PKg8jXzN8BxZWrR83/cKJSZ34jViMN6nan jI9w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=NsOkT0Cuq1Y0XzPa9XKC2sgmpa2tm9deyAWtcg8gRZ8=; b=lVO/Wy4fZV43tbWp0Pm8uj0dY34bfa6FXc55A1WpKqm2QobGJiyF2IV+cYt4bAhjVb 8lvxsoAic2z9pnEFTB8sPBiyMXz/s1JCaCIaepsTLvo+7BLC3KRWcL0ClgK+Ww1DFUES CcxB4FcigFkCD+VFS//KfcP1Lr/ET2YHaMffbULpKAF2hGJV8zq9YUMY+fZayVVIMibl UVjEFGFBzP32LqJsSnzNTOBc6s/EXG1aqrGszvcR8Dva1xlfm2MdedxY3HU1nWJ67OWO SFnLN9gzQW4/sUFqWm1MnrT16ZqoIqdTuYz5i86t0KoR0DGlC8+CgWuF6wkMleW1Uzqs K/gQ== X-Gm-Message-State: AA6/9RlQ4E06kdq0jsa84Aq+Fha66gTEEAvBSxQF1ORwSOMMkBMhWKkCQd8p4aXEOLiTxMwK4dR6iD/jQiXWzkVn X-Received: by 10.25.190.69 with SMTP id o66mr4120919lff.109.1476871401744; Wed, 19 Oct 2016 03:03:21 -0700 (PDT) MIME-Version: 1.0 Received: by 10.25.92.219 with HTTP; Wed, 19 Oct 2016 03:03:20 -0700 (PDT) In-Reply-To: References: From: Fahar Abbas Date: Wed, 19 Oct 2016 15:03:20 +0500 Message-ID: Subject: Re: RM1849: Auto-generating security keys To: Dave Page Cc: Ashesh Vashi , pgadmin-hackers , Josh Berkus , =?UTF-8?B?RGV2cmltIEfDnE5Ew5xa?= , Magnus Hagander , Sandeep Thakkar , Hamid Quddus Akhtar Content-Type: multipart/alternative; boundary=94eb2c1a1e0a47bdb5053f34eb93 X-Pg-Spam-Score: -2.6 (--) List-Archive: List-Help: List-ID: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: X-Mailing-List: pgadmin-hackers Precedence: bulk Sender: pgadmin-hackers-owner@postgresql.org --94eb2c1a1e0a47bdb5053f34eb93 Content-Type: text/plain; charset=UTF-8 Dave, Testing Environment Ubuntu 16.04 Linux 64: -------------------------------- pg-AdminIV Development Environment Setup for Ubuntu : 1) Install GIT sudo apt-get install git 2) Install pip3 sudo apt-get install python3-pip 3) Install virtualenv sudo pip3 install virtualenv 4) install below dependency as it is required for psycopg2 & pycrypto module sudo apt-get install libpq-dev sudo apt-get install python3-dev 5) Create virtual environment virtualenv -p python3 venv 6) Create mkdir Projects 7) Clone git repo in Projects git clone http://git.postgresql.org/git/pgadmin4.git 8) activate virtual environment source venv/bin/activate 9) Install modules pip3 install -r requirements_py3.txt *10) Edit the config.py file to config_local.py resides in Projects\pgAdmin4\web * 11)Now run setup.py file (\Projects\pgAdmin4\web) python setup.py If user does not create config_local.py and do Python setup.py for new Development then SECURITY_PASSWORD_SALT message is also displayed: Here is the output: ------------------------- python setup.py pgAdmin 4 - Application Initialisation ====================================== The configuration database - '/home/fahar/.pgadmin/pgadmin4.db' does not exist. Entering initial setup mode... NOTE: Configuring authentication for SERVER mode. Enter the email address and password to use for the initial pgAdmin user account: Email address: fahar.abbas@enterprisedb.com Password: Retype password: Traceback (most recent call last): File "setup.py", line 449, in do_setup(app) File "setup.py", line 96, in do_setup password = encrypt_password(p1) File "/home/fahar/venv/lib/python3.5/site-packages/flask_security/utils.py", line 150, in encrypt_password signed = get_hmac(password).decode('ascii') File "/home/fahar/venv/lib/python3.5/site-packages/flask_security/utils.py", line 108, in get_hmac 'set to "%s"' % _security.password_hash) RuntimeError: The configuration value `SECURITY_PASSWORD_SALT` must not be None when the value of `SECURITY_PASSWORD_HASH` is set to "pbkdf2_sha512" (venv) fahar@fahar-virtual-machine:~/Projects/pgadmin4/web$ Is this expected? On Wed, Oct 19, 2016 at 1:37 PM, Fahar Abbas wrote: > Sure, > > Will test this thoroughly after complete investigation. > > Kind Regards, > > On Wed, Oct 19, 2016 at 1:27 PM, Dave Page wrote: > >> Patch applied. >> >> Fahar, can you please test this thoroughly in desktop and server modes, >> with both fresh and upgraded installations? >> >> https://redmine.postgresql.org/issues/1849 >> >> Packagers: This change means that packages are no longer forced to create >> a config_local.py file, and there is no longer any need to explicitly set >> SECURITY_PASSWORD_SALT, SECURITY_KEY and CSRF_SESSION_KEY in the config >> (in fact, they should be removed for new installations, if you have >> included them in 1.0) >> >> Thanks. >> >> >> On Wed, Oct 19, 2016 at 6:46 AM, Ashesh Vashi < >> ashesh.vashi@enterprisedb.com> wrote: >> >>> Hi Dave, >>> >>> On Sat, Oct 15, 2016 at 8:02 AM, Dave Page wrote: >>> >>>> Hi >>>> >>>> >>>> On Friday, October 14, 2016, Dave Page wrote: >>>> >>>>> Hi >>>>> >>>>> On Thursday, October 13, 2016, Ashesh Vashi < >>>>> ashesh.vashi@enterprisedb.com> wrote: >>>>> >>>>>> Hi Dave, >>>>>> >>>>>> On Tue, Oct 11, 2016 at 9:10 PM, Dave Page wrote: >>>>>> >>>>>>> Hi Ashesh, >>>>>>> >>>>>>> Can you please review the attached patch, and apply if you're happy >>>>>>> with it? >>>>>>> >>>>>> Overall the patch looked good to me. >>>>>> But - I encounter an issue in 'web' mode, which wont happen with >>>>>> 'runtime'. >>>>>> >>>>>> Steps for reproduction on existing pgAdmin 4 environment with 'web' >>>>>> mode. >>>>>> - Apply the patch >>>>>> - Start the pgAdmin4 application (stand alone application). >>>>>> - Open pgAdmin home page. >>>>>> - Log out (if already login). >>>>>> >>>>>> And, you will see an exception. >>>>>> >>>>>> I have figure out the issue with the patch. >>>>>> We were setting the SECURITY_PASSWORD_SALT, after initializing the >>>>>> Security object. >>>>>> Hence - it could not set the SECURITY_KEY, and SECURITY_PASSWORD_SALT >>>>>> properly. >>>>>> >>>>> >>>>> Hmm. >>>>> >>>>> >>>>>> >>>>>> I had moved the Security object initialization after fetching these >>>>>> configurations from the database. >>>>>> I have attached a addon patch for the same. >>>>>> >>>>> >>>>> OK, thanks. >>>>> >>>>> >>>>>> >>>>>> Now - I run into another issue. >>>>>> Because - the existing password was hashed using the old >>>>>> SECURITY_PASSWORD_SALT, I am no more able to login to pgAdmin 4. >>>>>> >>>>>> I think - we need to think about different strategy for upgrading the >>>>>> configuration file in the 'web' mode. >>>>>> I was thinking - we can store the existing security configurations in >>>>>> the database during upgrade process in 'web' mode. >>>>>> >>>>> >>>>> My concern with that is that we'll likely be storing the default >>>>> config values in many cases, thus for those users, perpetuating the problem. >>>>> >>>>> I guess what we need to do is re-encrypt the password during the >>>>> upgrade - however, that makes me think; we then have both the key and the >>>>> encrypted passwords in the same database which is clearly not a good idea. >>>>> Sigh... Needs more thought. >>>>> >>>> >>>> OK, so I've been thinking about this and experimenting for a couple of >>>> hours, as well as annoying the crap out of Magnus by thinking out loud in >>>> his general direction, and it looks like this isn't a major problem as from >>>> what I can see, SECURITY_PASSWORD_SALT is (aside from really being a key >>>> not a salt) not the only salting that's done. >>>> >>>> It looks like it's used system-wide as the key to generate an HMAC of >>>> the users password, which is then passed to passlib which salts and hashes >>>> it. I did some testing, and found that two users with the same password end >>>> up with different hashes in the database, so clearly there is also per-user >>>> salting happening. I also created two users, then dropped the database and >>>> created the same user accounts with the same passwords again, and found >>>> that the resulting hashes were different in both databases - thus there is >>>> something else ensuring the hashes are unique across different >>>> installations/databases. >>>> >>>> So, I believe we can do as you suggest and migrate existing values for >>>> SECURITY_PASSWORD_SALT, given that there's clearly some other per user and >>>> per installation/database salting going on anyway. New installations can >>>> have the random value for SECURITY_PASSWORD_SALT. >>>> >>> We do not need to generate the random SECURITY_PASSWORD_SALT during >>> upgrade mode, which was wrong added in my addon patch. >>> >>> Please find the updated patch. >>> >>> Otherwise - looks good to me. >>> Please commit the new patch (if you're ok with the change). >>> >>> >>> -- >>> >>> Thanks & Regards, >>> >>> Ashesh Vashi >>> EnterpriseDB INDIA: Enterprise PostgreSQL Company >>> >>> >>> >>> *http://www.linkedin.com/in/asheshvashi* >>> >>> >>>> >>>> I don't believe SECURITY_KEY and CSRF_SESSION_KEY are issues either, as >>>> they're used for purposes that are essentially ephemeral, and thus can be >>>> changed during an upgrade. >>>> >>>> Adding Magnus as I'd appreciate any thoughts he may have. >>>> >>>> Patch attached - please review (Ashesh, but others too would be >>>> appreciated)! >>>> >>>> Thanks. >>>> >>>> >>>> -- >>>> Dave Page >>>> Blog: http://pgsnake.blogspot.com >>>> Twitter: @pgsnake >>>> >>>> EnterpriseDB UK: http://www.enterprisedb.com >>>> The Enterprise PostgreSQL Company >>>> >>>> >>> >> >> >> -- >> Dave Page >> Blog: http://pgsnake.blogspot.com >> Twitter: @pgsnake >> >> EnterpriseDB UK: http://www.enterprisedb.com >> The Enterprise PostgreSQL Company >> > > > > -- > Syed Fahar Abbas > Quality Management Group > > EnterpriseDB Corporation > Phone Office: +92-51-835-8874 > Phone Direct: +92-51-8466803 > Mobile: +92-333-5409707 > Skype ID: syed.fahar.abbas > Website: www.enterprisedb.com > -- Syed Fahar Abbas Quality Management Group EnterpriseDB Corporation Phone Office: +92-51-835-8874 Phone Direct: +92-51-8466803 Mobile: +92-333-5409707 Skype ID: syed.fahar.abbas Website: www.enterprisedb.com --94eb2c1a1e0a47bdb5053f34eb93 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Dave,

Testing Environment
<= /div>=C2=A0
Ubuntu 16.04 Linux 64:
----------------------------= ----

pg-AdminIV Develop= ment Environment Setup for Ubuntu=C2=A0 :


1) Install GIT

sudo apt-get install git

2) Install pip3

sudo apt-get install python3-pip

3) Install virtualenv=

sudo pip3 install = virtualenv

4= ) install below dependency as it is required for psycopg2 & pycrypto mo= dule

sudo ap= t-get install libpq-dev

sudo apt-get install python3-dev

5) Create virtual environment

virtualenv -p python3= venv

6) Create mkdir Pr= ojects

7= ) Clone git repo in Projects

git clone http://git.postgresql.org/git/pgadmin4.git

8) activate virtual environment

source venv/bin/activate

9) Install modu= les

pip3 ins= tall -r requirements_py3.txt

10) Edit the config.py fil= e to config_local.py =C2=A0resides in Projects\pgAdmin4\web=C2=A0=C2=A0

11)Now run setup= .py file =C2=A0(\Projects\pgAdmin4\web)

=C2=A0 =C2=A0 python setup= .py

If user does not create config_local.py and do P= ython setup.py for new Development then SECURITY_PASSWORD_SALT message is a= lso displayed:

Here is the output:
-------------------= ------

python setup.py
pgAdmin 4 - Application Initia= lisation
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D


The configur= ation database - '/home/fahar/.pgadmin/pgadmin4.db' does not exist.=
Entering initial setup mode...
NOTE: Configuring authentication for = SERVER mode.


=C2=A0=C2=A0=C2=A0 Enter the email address and pass= word to use for the initial pgAdmin user=C2=A0=C2=A0=C2=A0=C2=A0 account:
Email address: fahar.= abbas@enterprisedb.com
Password:
Retype password:
Traceback (= most recent call last):
=C2=A0 File "setup.py", line 449, in &= lt;module>
=C2=A0=C2=A0=C2=A0 do_setup(app)
=C2=A0 File "setu= p.py", line 96, in do_setup
=C2=A0=C2=A0=C2=A0 password =3D encrypt= _password(p1)
=C2=A0 File "/home/fahar/venv/lib/python3.5/site-pack= ages/flask_security/utils.py", line 150, in encrypt_password
=C2=A0= =C2=A0=C2=A0 signed =3D get_hmac(password).decode('ascii')
=C2= =A0 File "/home/fahar/venv/lib/python3.5/site-packages/flask_security/= utils.py", line 108, in get_hmac
=C2=A0=C2=A0=C2=A0 'set to &qu= ot;%s"' % _security.password_hash)
RuntimeError: The configurat= ion value `SECURITY_PASSWORD_SALT` must not be None when the value of `SECU= RITY_PASSWORD_HASH` is set to "pbkdf2_sha512"
(venv) fahar@fah= ar-virtual-machine:~/Projects/pgadmin4/web$


Is this expected?

On Wed, Oct 19, 20= 16 at 1:37 PM, Fahar Abbas <fahar.abbas@enterprisedb.com>= ; wrote:
Sure,

Will test this thoroughly after complete investigation= .

Kind Regards,

On Wed, Oct 19, 2016 at 1:27 PM= , Dave Page <dpage@pgadmin.org> wrote:
Patch applied.

Fahar, can= you please test this thoroughly in desktop and server modes, with both fre= sh and upgraded installations?


Packagers: This cha= nge means that packages are no longer forced to create a config_local.py fi= le, and there is no longer any need to explicitly set=C2=A0SECURITY_PASSWORD_SALT,=C2=A0SECURITY_KEY and=C2=A0CSRF_SESSION_KEY in the config (in fa= ct, they should be removed for new installations, if you have included them= in 1.0)

Thanks.


On Wed, Oct 19, 2016 at 6:46 AM, Ashesh Vashi <ashesh.vas= hi@enterprisedb.com> wrote:
Hi Dave,

On Sat, Oct 15, 2016 at 8:02 AM, D= ave Page <dpage@pgadmin.org> wrote:
Hi


On Friday, = October 14, 2016, Dave Page <dpage@pgadmin.org> wrote:
Hi

On Thursday, October 13, 2016, Ashesh Vash= i <ashesh.vashi@enterprisedb.com> wrote:
Hi Dave,

On Tue, Oct 11, 201= 6 at 9:10 PM, Dave Page <dpage@pgadmin.org><= /span> wrote:
Hi Ashes= h,

Can you please review the attached patch, and apply i= f you're happy with it?
Overall the patch looked= good to me.
But - I encounter an issue in 'web' mode, wh= ich wont happen with 'runtime'.

Steps for = reproduction on existing pgAdmin 4 environment with 'web' mode.
- Apply the patch
- Start the pgAdmin4 application (stand = alone application).
- Open pgAdmin home page.
- Log out= (if already login).

And, you will see an exceptio= n.

I have figure out the issue with the patch.
We were setting the SECURITY_PASSWORD_SALT, after initializing the S= ecurity object.
Hence - it could not set the SECURITY_KEY, and SE= CURITY_PASSWORD_SALT properly.
Hmm.
=C2=A0

I had moved the Security object initializatio= n after fetching these configurations from the database.
I have a= ttached a addon patch for the same.

OK, thanks.
=C2=A0

Now - I run into another issue.
Because - the existing password was hashed using the old SECURITY_= PASSWORD_SALT, I am no more able to login to pgAdmin 4.

I think - we need to think about different strategy for upgrading the= configuration file in the 'web' mode.
I was thinking - w= e can store the existing security configurations in the database during upg= rade process in 'web' mode.

My concern with that is that we'll likely be storing t= he default config values in many cases, thus for those users, perpetuating = the problem.

I guess what we need to do is re-encr= ypt the password during the upgrade - however, that makes me think; we then= have both the key and the encrypted passwords in the same database which i= s clearly not a good idea. Sigh... Needs more thought.=C2=A0

OK, so I've been thinking about thi= s and experimenting for a couple of hours, as well as annoying the crap out= of Magnus by thinking out loud in his general direction, and it looks like= this isn't a major problem as from what I can see, =C2=A0SECURITY_PASS= WORD_SALT is (aside from really being a key not a salt) not the only saltin= g that's done.=C2=A0

It looks like it's us= ed system-wide as the key to generate an HMAC of the users password, which = is then passed to passlib which salts and hashes it. I did some testing, an= d found that two users with the same password end up with different hashes = in the database, so clearly there is also per-user salting happening. I als= o created two users, then dropped the database and created the same user ac= counts with the same passwords again, and found that the resulting hashes w= ere different in both databases - thus there is something else ensuring the= hashes are unique across different installations/databases.

=
So, I believe we can do as you suggest and migrate existing valu= es for SECURITY_PASSWORD_SALT, given that there's clearly some other pe= r user and per installation/database salting going on anyway. New installat= ions can have the random value for SECURITY_PASSWORD_SALT.
We do not need to generate the random SECURITY_PASSWORD_= SALT during upgrade mode, which was wrong added in my addon patch.

Please find the updated patch.

Ot= herwise - looks good to me.
Please commit the new patch (if you&#= 39;re ok with the change).


--

<= span class=3D"m_-4019292325802352314m_2436846264967577571gmail-">Thanks & Regards,

= Ashesh Vashi
EnterpriseDB INDIA:=C2=A0Enterprise PostgreSQL Company



I don't believe SECURITY_KE= Y and=C2=A0CSRF_SESSION_KEY are issues either, as they're used for purp= oses that are essentially ephemeral, and thus can be changed during an upgr= ade.

Adding Magnus as I'd appreciate any thoug= hts he may have.

Patch attached - please review (A= shesh, but others too would be appreciated)!

Thank= s.


--
Dave Page<= br>Blog: http://p= gsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb= .com
The Enterprise PostgreSQL Company





--
Dave PageBlog: http://pg= snake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.= com
The Enterprise PostgreSQL Company



--
Syed= Fahar Abbas
Quality Management Group

Enterprise= DB Corporation
Phone Office: +92-51-835-8874
Phone Direct: +92-51-8466= 803
Mobile: +92-333-5409707
Skype ID: syed.fahar.abbas
We= bsite: www.enterp= risedb.com


--
Syed Fahar= Abbas
Quality Management Group

EnterpriseDB Cor= poration
Phone Office: +92-51-835-8874
Phone Direct: +92-51-8466803Mobile: +92-333-5409707
Skype ID: syed.fahar.abbas
Website: www.enterprisedb.com=
--94eb2c1a1e0a47bdb5053f34eb93--