Subject: Re: TLS certificate alternate trust paths issue in libpq - certificate chain validation failing

From: Thomas Spear <[email-masked]>
Date: Wed, 1 May 2024 10:17:13 -0500

On Wed, May 1, 2024 at 9:23 AM Jacob Champion <
[email-masked]> wrote:

> On Wed, May 1, 2024 at 6:48 AM Thomas Spear <[email-masked]> wrote:
> > I dumped out the certificates presented by the server using openssl, and
> the chain that gets output includes "Microsoft Azure RSA TLS Issuing CA 08".
> > On https://www.microsoft.com/pkiops/docs/repository.htm the page says
> that that cert was cross-signed by the DigiCert RSA G2 root.
>
> It's been a while since I've looked at cross-signing, but that may not
> be enough information to prove that it's the "correct" version of the
> intermediate. You'd need to know the Issuer, not just the Subject, for
> all the intermediates that were given to the client. (It may not match
> the one they have linked on their support page.)
>
>
Fair enough. The server issuer is C=US,O=Microsoft Corporation,CN=Microsoft
Azure RSA TLS Issuing CA 08
The intermediate's issuer is C=US,O=Microsoft Corporation,CN=Microsoft RSA
Root Certificate Authority 2017 so I think that you're absolutely correct.
The intermediate on the support page reflects the DigiCert issuer, but the
one from the server reflects the Microsoft issuer.

Circling back to my original question, why is there a difference in
behavior?

What I believe should be happening isn't what's happening:
1. If ~/.postgresql/root.crt contains the MS root, and I don't specify
sslrootcert= -- successful validation
2. If ~/.postgresql/root.crt contains the MS root, and I specify
sslrootcert= -- successful validation
3. If ~/.postgresql/root.crt contains the DigiCert root, and I don't
specify sslrootcert= -- validation fails
4. If ~/.postgresql/root.crt contains the DigiCert root, and I specify
sslrootcert= -- successful validation

Case 3 should succeed IMHO since case 4 does.

> > The postgres server appears to send the Microsoft root certificate
> instead of the DigiCert one, which should be fine. The server sends the
> "Microsoft RSA Root Certificate Authority 2017" root.
> > As far as I understand, a server sending a root certificate along with
> the intermediate is a big no-no, but that's a topic for a different thread
> and audience most likely. :)
>
> To me, that only makes me more suspicious that the chain the server is
> sending you may not be the chain you're expecting. Especially since
> you mentioned on the other thread that the MS root is working and the
> DigiCert root is not.
>
>
Yeah, I agree. So then I need to talk to MS about why the portal is giving
us the wrong root -- and I'll open a support ticket with them for this. I
still don't understand why the above difference in behavior happens though.
Is that specifically because the server is sending the MS root? Still
doesn't seem to make a whole lot of sense. If the DigiCert root can
validate the chain when it's explicitly passed, it should be able to
validate the chain when it's implicitly the only root cert available to a
postgres client.

> > The openssl version in my Windows test system is 3.0.7. It's running
> Almalinux 9 in WSL2, so openssl is from the package manager. The container
> image I'm using has an old-as-dirt openssl 1.1.1k.
>
> I'm not aware of any validation issues with 1.1.1k, for what it's
> worth. If upgrading helps, great! -- but I wouldn't be surprised if it
> didn't.
>
>
I was thinking the same honestly. If it breaks for jdbc-postgres on 1.1.1k
and psql cli on 3.0.7 then it's likely not an issue there.

> I'll have to check one of our public cloud postgres instances to see if I
> can reproduce the issue there in order to get a chain that I can share
> because the system where I'm testing is a locked down jump host to our
> Azure GovCloud infrastructure, and I can't copy anything out from it.
>
> Yeah, if at all possible, that'd make it easier to point at any
> glaring problems.
>
>
I should be able to do this today.

Thanks again!

--Thomas

---

From: Jacob Champion <[email-masked]>
Date: Wed, 1 May 2024 10:30:56 -0700

On Wed, May 1, 2024 at 8:17 AM Thomas Spear <[email-masked]> wrote:
> Circling back to my original question, why is there a difference in behavior?
>
> 1. If ~/.postgresql/root.crt contains the MS root, and I don't specify sslrootcert= -- successful validation
> 2. If ~/.postgresql/root.crt contains the MS root, and I specify sslrootcert= -- successful validation
> 3. If ~/.postgresql/root.crt contains the DigiCert root, and I don't specify sslrootcert= -- validation fails
> 4. If ~/.postgresql/root.crt contains the DigiCert root, and I specify sslrootcert= -- successful validation

Number 4 is the only one that seems off to me given what we know. If
you're saying that the server's chain never mentions DigiCert as an
issuer, then I see no reason that the DigiCert root should ever
successfully validate the chain. You mentioned on the other thread
that

> We eventually found the intermediate cert was missing from the system trust, so we tried adding that without success

and that has me a little worried. Why would the intermediate need to
be explicitly trusted?

I also notice from the other thread that sometimes you're testing on
Linux and sometimes you're testing on Windows, and that you've mixed
in a couple of different sslmodes during debugging. So I want to make
absolutely sure: are you _certain_ that case number 4 is a true
statement? In other words, there's nothing in your default root.crt
except the DigiCert root, you've specified exactly the same path in
sslrootcert as the one that's loaded by default, and your experiments
are all using verify-full?

The default can also be modified by a bunch of environmental factors,
including $PGSSLROOTCERT, $HOME, the effective user ID, etc. (On
Windows I don't know what the %APPDATA% conventions are.) If you empty
out your root.crt file, you should get a clear message that libpq
tried to load certificates from it and couldn't; otherwise, it's
finding the default somewhere else.

--Jacob