Subject: Re: Proposal: Support custom authentication methods using hooks

From: Tom Lane <[email-masked]>
Date: Fri, 25 Feb 2022 14:10:39 -0500

Jeff Davis <[email-masked]> writes:
> On Fri, 2022-02-25 at 12:39 -0500, Tom Lane wrote:
>> My point is that sending cleartext passwords over the wire is an
>> insecure-by-definition protocol that we shouldn't be encouraging
>> more use of.

> We can require custom auth entries in pg_hba.conf to also specify
> local, hostssl or hostgssenc.

That's just a band-aid, though.  It does nothing for the other
reasons not to want cleartext passwords, notably that you might
be giving your password to a compromised server.

I'm happy to add support for custom auth methods if they can use
a protocol that's safer than cleartext-password.  But if that's the
only feasible option, then we're just encouraging people to use
insecure methods.

I also have in mind here that there has been discussion of giving
libpq a feature to refuse, on the client side, to send cleartext
passwords.  (I don't recall right now if that's been mentioned
publicly or only among the security team, but it's definitely
under consideration.)

So I think this proposal needs more thought.  A server-side hook
alone is not a useful improvement IMO; it's closer to being an
attractive nuisance.

			regards, tom lane

---

From: Andres Freund <[email-masked]>
Date: Fri, 25 Feb 2022 11:49:02 -0800

Hi,

> only feasible option, then we're just encouraging people to use

It looks like scram can be used without much trouble. All the necessary
infrastructure to implement it without duplication appears to be public.  The
plugin would need to get a secret from whereever and call
  CheckSASLAuth(&pg_be_scram_mech, port, shadow_pass, logdetail);

or if it needs to do something more complicated than CheckSASLAuth(), it can
use AUTH_REQ_SASL{,_FIN CONT} itself.

Of course whether it's viable depends on what the custom auth method wants to
do. But it's not a restriction of the authentication plugin model.

Greetings,

Andres Freund

---

From: Jeff Davis <[email-masked]>
Date: Sun, 27 Feb 2022 15:17:44 -0800

> only feasible option, then we're just encouraging people to use

FWIW, I'd like to be able to use a token in the password field, and
then authenticate that token. So I didn't intend to send an actual
password in plaintext.

Maybe we should have a new "token" connection parameter so that libpq
can allow sending a token plaintext but refuse to send a password in
plaintext?

> passwords.

I am generally in favor of that idea, but I'm not sure that will
completely resolve the issue. For instance, should it also refuse MD5?

Regards,
	Jeff Davis

---

From: Stephen Frost <[email-masked]>
Date: Mon, 28 Feb 2022 15:46:34 -0500

Greetings,

> completely resolve the issue. For instance, should it also refuse MD5?

md5 should be removed.

Thanks,

Stephen

---

From: Tom Lane <[email-masked]>
Date: Mon, 28 Feb 2022 16:18:48 -0500

> md5 should be removed.

Really? I've always thought that the arguments against it were
overblown for our use-case.  At any rate, it's likely to be
years before we could practically do that, since it's the best
that older client libraries can manage.

			regards, tom lane

---

From: Stephen Frost <[email-masked]>
Date: Mon, 28 Feb 2022 16:42:55 -0500

Greetings,

> that older client libraries can manage.

Yes, really, it's a known-broken system which suffers from such an old
and well known attack that it's been given a name: pass-the-hash.  As
was discussed on this thread even, just the fact that it's not trivial
to break on the wire doesn't make it not-broken, particularly when we
use the username (which is rather commonly the same one used across
multiple systems..) as the salt.  Worse, md5 isn't exactly the pinnacle
of hashing techniques around these days.

The wiki page goes over it in some detail regarding LM/NTLM which
suffers the same problem (and also uses a challenge-response for the
over-the-network bits): https://en.wikipedia.org/wiki/Pass_the_hash

Further, a whole bunch of effort was put in to get scram support added
to the different libraries and language bindings and such, specifically
to allow us to get to a point where we can drop md5.  Even after it's
removed, folks will have 5 years before the release that removes it is
the oldest supported release.  I don't think we'll somehow get agreement
to remove it for v15, so it'll be 5 major versions of overlap (11->15)
by the time v16 comes out, and a total of 10 years of support for scram
before md5 is gone.

That's plenty, it's time to move on.

Keeping it around will just push out the point at which everyone will
finally be done with it, as there's really only two groups: those who
have already moved to scram, and those who won't move until they want to
upgrade to a release that doesn't have md5.

Thanks,

Stephen

---

From: Michael Paquier <[email-masked]>
Date: Tue, 1 Mar 2022 15:26:23 +0900

> upgrade to a release that doesn't have md5.

FWIW, I am not sure if we are at this point yet.  An extra reason to
remove it would be that it is a support burden, but I don't have seen
in recent memory any problems related to it that required any deep
changes in the way to use it, and its code paths are independent.

The last time I played with this area is the recent error handling
improvement with cryptohashes but MD5 has actually helped here in
detecting the problem as a patched OpenSSL would complain if trying to
use MD5 as hash function when FIPS is enabled.
--
Michael

---

From: Stephen Frost <[email-masked]>
Date: Tue, 1 Mar 2022 08:31:19 -0500

Greetings,

> changes in the way to use it, and its code paths are independent.

Ongoing reports that there's a known vulnerability aren't great to have
to deal with.  We can at least point people to scram but that's not
great.

> use MD5 as hash function when FIPS is enabled.

Having to continue to deal with md5 as an algorithm when it's known to
be notably less secure and so much so that organizations essentially ban
its use for exactly what we're using it for, in fact, another reason to
remove it, not a reason to keep it.  Better code coverage testing of
error paths is the answer to making sure that our error handling behaves
properly.

Thanks,

Stephen

---

From: Nathan Bossart <[email-masked]>
Date: Tue, 1 Mar 2022 13:14:26 -0800

> md5 should be removed.

+1

-- 
Nathan Bossart
Amazon Web Services: https://aws.amazon.com

---

From: "Jonathan S. Katz" <[email-masked]>
Date: Tue, 1 Mar 2022 16:17:28 -0500

> error paths is the answer to making sure that our error handling behaves

So, I generally agree with Stephen and his arguments for dropping the 
md5 mechanism.

If you're moving to a newer version of PostgreSQL, you likely have to 
update your connection drivers anyway (rebuilt against new libpq, 
supporting any changes in the protocol, etc). I would prefer more data 
to support that argument, but this is generally what you need to do.

However, we may need to step towards it. We took one step last release 
with defaulting to SCRAM. Perhaps this release we add a warning for 
anything using md5 auth that "this will be removed in a future release." 
(or specifically v16). We should also indicate in the docs that md5 is 
deprecated and will be removed.

We also need to think about upgrading: what happens if I try to upgrade 
and I still have user passwords stored in a md5 hash? What if all my 
password-based users have a md5 hash, does pg_upgrade fail?

As much as I want md5 gone, I think v16 is the earliest we can do it, 
but we can lay the groundwork for it now.

Thanks,

Jonathan

---

From: Peter Eisentraut <[email-masked]>
Date: Wed, 2 Mar 2022 09:24:26 +0100

> deprecated and will be removed.

I find that a lot of people are still purposely using md5.  Removing it 
now or in a year would be quite a disruption.

It's also worth considering that keeping the code equipped to handle 
different kinds of password hashing would help it stay in shape if we 
ever need to add support for the next SHA after 256 or whatever.

---

From: "Jonathan S. Katz" <[email-masked]>
Date: Wed, 2 Mar 2022 09:16:10 -0500

On 3/2/22 3:24 AM, Peter Eisentraut wrote:
>> anything using md5 auth that "this will be removed in a future 
>> release." (or specifically v16). We should also indicate in the docs 
>> that md5 is deprecated and will be removed.
> 

What are the reasons they are still purposely using it? The ones I have 
seen/heard are:

- Using an older driver
- On a pre-v10 PG
- Unaware of SCRAM

What I'm proposing above is to start the process of deprecating it as an 
auth method, which also allows to continue the education efforts to 
upgrae. Does that make sense?

> ever need to add support for the next SHA after 256 or whatever.

I think it's fine to keep the hashing code. The end goal is to remove 
the md5 authentication mechanism.

Thanks,

Jonathan

---

From: Bruce Momjian <[email-masked]>
Date: Wed, 2 Mar 2022 09:55:15 -0500

> its use for exactly what we're using it for, in fact, another reason to

Really?  I thought it was publicly-visible MD5 hashes that were the
biggest problem.  Our 32-bit salt during the connection is a problem, of
course.

> error paths is the answer to making sure that our error handling behaves

What is the logic to removing md5 but keeping 'password'?

-- 
  Bruce Momjian  <[email-masked]>        https://momjian.us
  EDB                                      https://enterprisedb.com

  If only the physical world exists, free will is an illusion.

---

From: Stephen Frost <[email-masked]>
Date: Wed, 2 Mar 2022 09:58:41 -0500

Greetings,

> biggest problem.  Our 32-bit salt during the connection is a problem, of

Neither are good.  Not sure that we really need to spend a lot of effort
trying to figure out which issue is the biggest problem.

> What is the logic to removing md5 but keeping 'password'?

I don't think we should keep 'password'.

Thanks,

Stephen

---

From: Tom Lane <[email-masked]>
Date: Wed, 02 Mar 2022 10:01:59 -0500

> I don't think we should keep 'password'.

I don't see much point in that unless we deprecate *all* the
auth methods that transmit a cleartext password.

			regards, tom lane

---

From: Peter Eisentraut <[email-masked]>
Date: Wed, 2 Mar 2022 16:29:29 +0100

On 02.03.22 15:16, Jonathan S. Katz wrote:
>> I find that a lot of people are still purposely using md5.  Removing 
>> it now or in a year would be quite a disruption.
> 

I'm not really sure, but it seems like they are content with what they 
have and don't want to bother with the new fancy stuff.

> upgrae. Does that make sense?

I'm not in favor of starting a process that will result in removal of 
the md5 method at this time.

---

From: Stephen Frost <[email-masked]>
Date: Wed, 2 Mar 2022 10:30:52 -0500

Greetings,

* Peter Eisentraut ([email-masked]) wrote:
> I'm not really sure, but it seems like they are content with what they have
> and don't want to bother with the new fancy stuff.

There were lots and lots of folks who were comfortable with
recovery.conf, yet we removed that without any qualms from one major
version to the next.  md5 will have had 5 years of overlap with scram.

> I'm not in favor of starting a process that will result in removal of the
> md5 method at this time.

I am.

Thanks,

Stephen

---

From: "Jonathan S. Katz" <[email-masked]>
Date: Wed, 2 Mar 2022 10:45:01 -0500

>> and don't want to bother with the new fancy stuff.

By that argument, we should have kept "password" (plain) as an 
authentication method.

The specific use-cases I've presented are all solvable issues. The 
biggest challenging with existing users is the upgrade process, which is 
why I'd rather we begin a deprecation process and see if there are any 
ways we can make the md5 => SCRAM transition easier.

> version to the next.  md5 will have had 5 years of overlap with scram.

I do agree with Stephen in principle here. I encountered upgrade 
challenges in this an challenge with updating automation to handle this 
change.

>> I'm not in favor of starting a process that will result in removal of the

+1 for starting this process. It may still take a few more years, but we 
should help our users to move away from an auth method with known issues.

Thanks,

Jonathan

---

From: Tatsuo Ishii <[email-masked]>
Date: Thu, 03 Mar 2022 15:15:24 +0900 (JST)

> multiple systems..) as the salt.  Worse, md5 isn't exactly the pinnacle

I am not a big fan of md5 auth but saying that md5 auth uses username
as the salt is oversimplified. The md5 hashed password shored in
pg_shadow is created as md5(password + username).  But the md5 hashed
password flying over wire is using a random salt like md5(md5(password
+ username) + random_salt).

Best reagards,
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese:http://www.sraoss.co.jp

---

From: Peter Eisentraut <[email-masked]>
Date: Thu, 3 Mar 2022 10:45:42 +0100

> By that argument, we should have kept "password" (plain) as an 

For comparison, the time between adding md5 and removing password was 16 
years.  It has been 5 years since scram was added.

---

From: Peter Eisentraut <[email-masked]>
Date: Thu, 3 Mar 2022 10:59:55 +0100

> What are the reasons they are still purposely using it? The ones I have 

Another reason is that SCRAM presents subtle operational issues in 
distributed systems.  As someone who is involved with products such as 
pgbouncer and bdr, I am aware that there are still unresolved problems 
and ongoing research in that area.  Maybe they can all be solved 
eventually, even if it is concluding "you can't do that anymore" in 
certain cases, but it's not all solved yet, and falling back to the 
best-method-before-this-one is a useful workaround.

I'm thinking there might be room for an authentication method between 
plain and scram that is less complicated and allows distributed systems 
to be set up more easily.  I don't know what that would be, but I don't 
think we should prohibit the consideration of "anything less than SCRAM".

I notice that a lot of internet services are promoting "application 
passwords" nowadays.  I don't know the implementation details of that, 
but it appears that the overall idea is to have instead of one 
high-value password have many frequently generated medium-value 
passwords.  We also have a recent proposal to store multiple passwords 
per user.  (Obviously that could apply to SCRAM and not-SCRAM equally.) 
That's the kind of direction I would like to explore.

---

From: Joshua Brindle <[email-masked]>
Date: Thu, 3 Mar 2022 11:12:17 -0500

> years.  It has been 5 years since scram was added.

It's been 7 years since this thread:
https://www.postgresql.org/message-id/[email-masked]

As Jonathan and Stephen and others have said, anyone who wishes to
continue using MD5 or other plaintext methods can keep doing that for
5 more years with a supported version of PG. There is no excuse to
leave well known, flawed mechanisms in PG16.

---

From: Stephen Frost <[email-masked]>
Date: Thu, 3 Mar 2022 12:31:05 -0500

Greetings,

> + username) + random_salt).

Err, no, it's not oversimplified at all- we do, in fact, as you say
above, use the username as the salt for what gets stored in pg_authid
(pg_shadow is just a view).  That's absolutely a problem because servers
can be compromised, backups can be compromised, and when it comes to PG
servers you don't even need to actually bother cracking the password
once you've gained access to an md5 value in pg_authid anyway.

Yes, we do use a challenge/response over the wire but that doesn't
absolve us of the fact that the hashes we store in pg_authid with the
md5 method is subject to pass-the-hash and brute-force attacks against
it.  If anything, the challenge/response over the wire is less useful
considering the common usage of TLS these days.

Thanks,

Stephen