Subject: apt.postgresql.org repo via https will fail will some users starting 2021-10-01

From: Stefan Huehner <[email-masked]>
Date: Wed, 8 Sep 2021 16:48:06 +0000

Hello,

sending this here as looks like https://apt.postgresql.org is affected by this so this could trigger some support/user questions.

Note this only (!) happens when using https:// in sources.list for the pgdg repo.

Benefit of that is debatable (see recent debian-devel discussion) but i would not be surprised if some/many people use it.

Trigger:
https://community.letsencrypt.org/t/openssl-client-compatibility-changes-for-let-s-encrypt-certificates/143816

End of this month some CA cert will expire related to Let's Encrypt which will trigger an bug in clients using old openssl/gnutls.

apt is using gnutls backend and at least the version in Ubuntu <= 18.04 are affected and "apt update" will already fail for people starting that date.

Note that canonical is working in patching gnutls so if that finishes in time and (!) if people update before that date all good.

If not they will get error similar to:
Err:9 https://apt.postgresql.org/pub/repos/apt focal-pgdg Release               
  Certificate verification failed: The certificate is NOT trusted. The revocation or OCSP data are old and have been superseded. The certificate chain uses expired certificate.  Could not handshake: Error in the certificate verification. [IP: 87.238.57.227 443]

Can be triggered today i.e. with:

faketime "2021-10-01" apt update

Ideas:
- Do nothing apt.postgresql suggest http:// in the instructions
- Some on the website
- Think on reconfiguring certbot/Let's Encrypt on the server to switch to the alternative chain (avoiding this bug but breaking compatibility with old Android

- Raise as bug to debian also (against openssl/gnutls) to maybe patch both in stable also to avoid this ?
  - Not sure if that is a interesting/acceptable material for stable/old-stable?

Please CC me on replies i'm not on the list

Stefan

---

From: Christoph Berg <[email-masked]>
Date: Thu, 9 Sep 2021 14:33:49 +0200

Re: Stefan Huehner
> Note this only (!) happens when using https:// in sources.list for the pgdg repo.

Hi,

thanks for sharing this.

We aren't advertising https:// for apt.postgresql.org anywhere, but
the download instructions tell users to "wget" the repository key from
https://www.postgresql.org, so we are at least somewhat affected.
(wget is using gnutls at least in unstable.)

> - Think on reconfiguring certbot/Let's Encrypt on the server to switch to the alternative chain (avoiding this bug but breaking compatibility with old Android

That's probably rather the ca-certificates package?

>   - Not sure if that is a interesting/acceptable material for stable/old-stable?

If stretch/buster/bullseye are affected, these should be fixed, yes.

Though none of this is material for the PostgreSQL packages, can you
raise the issue with the LTS team?

Christoph

---

From: Stefan Huehner <[email-masked]>
Date: Thu, 9 Sep 2021 15:15:08 +0000

> That's probably rather the ca-certificates package?

Not in this case, i know a bit confusing.
That upstream article has more details:
https://community.letsencrypt.org/t/openssl-client-compatibility-changes-for-let-s-encrypt-certificates/143816
Part: How to support older OpenSSL versions

In (not so) short: ca-certificates is fine to have trust anchor for Lets Encrypt.
However not everybody directly trust Let's Encrypt (missing entry in their equivalent of ca-certificates (i.e. old Android).

To keep those other clients supported they employed a bit of a trick which has an 'expired root certificates' in the chain from your server-cert to their root. At the same time there is 2nd valid path. But old version of software (openssl,gnutls) just stop + fail on seeing 'expired'.

Best they could do if offer server owner (certbot parameter when requesting ssl certificate to select):
a.) Default chain (compatible still with old android) but triggering this bug
b.) Alternative chain (ignore old android) but keep compatible with old openssl/gnutls

That link goes into much more detail but hopefully now clearer.

That is also why i raised this here as a choice for apt.postgresql.org hosting (if you think it's a useful workaround)

> raise the issue with the LTS team?

Will raise there.

Hopefuly above also clarified why i sent that here (not about any PostgreSQL package, but apt.postgresql.org server admin topic).

Stefan

---

From: Christoph Berg <[email-masked]>
Date: Thu, 9 Sep 2021 17:33:51 +0200

Re: Stefan Huehner
> Best they could do if offer server owner (certbot parameter when requesting ssl certificate to select):

Ah, I thought you meant the end-users servers running PostgreSQL when
you said "server".

For changing the webservers, we'd need to get pginfra on board, Cc'ed
now.

Christoph

---

From: Stefan Huehner <[email-masked]>
Date: Thu, 9 Sep 2021 16:57:08 +0000

> Ah, I thought you meant the end-users servers running PostgreSQL when
Sorry for the confusion.

But now thinking they could be affected (but special cases only)
- There may be a case for running PostgreSQL instances
- As this affects any 'client' using above older libraries
- libpq linked with old ssl, connecting via SSL to remote pg having Let's Encrypt and client validating certificate (verify-ca,verify-full) 
- or outgoing connections using fdw 

Note:
This is just me trying to construct a flow which might fail.
Again kind of 'info for supporting users' as bug is libssl we link against only.
Maybe also interesting to spread to the yum-side of things (Devrim?) to check thei rpm using packaging/distros.
But also not sure if that is common enough even to make a big topic out of...

> For changing the webservers, we'd need to get pginfra on board, Cc'ed

For the sysadmins:
- 'changing' would avoid the bug described here + not for debian
  https://lists.debian.org/debian-lts/2021/09/msg00008.html
  for people running old distros (no fix, or not updated, ...)
- But it will break using the webserver with other clients (i.e. older android)
- Need to pick the 'smaller problem' based on the concrete site and its users/clients
- This is generic Let's Encrypt topic (not just apt.postgresql.org host)
- Above letsencrypt link has bigger explanation + all the background

Stefan

---

From: Christoph Berg <[email-masked]>
Date: Fri, 10 Sep 2021 10:50:36 +0200

Re: Stefan Huehner
> - As this affects any 'client' using above older libraries

On stretch and later, PG is linked against libssl1.1 which is safe, so
we are not affected. (jessie is EOL.)

Christoph

---

From: Magnus Hagander <[email-masked]>
Date: Tue, 14 Sep 2021 16:33:01 +0200

>   - Not sure if that is a interesting/acceptable material for stable/old-stable?

Hi!

We've started looking into what can and should be done on the infra
side to see if we can get this working.

One question though. In my attempts to reproduce, it seems that *wget*
on Ubuntu 18.04 has no problem with the current chain, just apt-get,
does that match with your testing?  So if one follows our instructions
of getting the gpg key with https but the actual repo with http, it
never actually presents a problem?

That's not saying we don't need to do anything about it, just to
reconfirm our tests. For example, this appears to also break RedHat 6
as well...

-- 
 Magnus Hagander
 Me: https://www.hagander.net/
 Work: https://www.redpill-linpro.com/

---

From: Stefan Huehner <[email-masked]>
Date: Fri, 17 Sep 2021 09:53:30 +0000

> never actually presents a problem?

Hi Magnus,

sorry for the delay i was out travelling.

I just checked and and wget for Ubuntu 18.04/bionic is linked to libssl1.1 not having the problem in the first place.
Which explains why is it not affected here. I think wget can have both gnutls+openssl backend so need to see which is used per distro/release (As can change)

# apt show wget | grep Depends
Depends: libc6 (>= 2.17), libidn2-0 (>= 0.6), libpcre3, libpsl5 (>= 0.16.0), libssl1.1 (>= 1.1.0), libuuid1 (>= 2.16)

Also for apt itself in 18.04 Ubuntu has backported the fix into gnutls30 and it just reached the normal update repository.

Package/version containing the fix is:
libgnutsl30	3.5.18-1ubuntu1.5

So here if people install (non-security) updates they will be not affected.
Or telling them "run system updates" if they get the bug is enough.

Note for older Ubuntu 16.04+14.04 Ubuntu offers paid updates still which are not yet done /published for this issue but also in progress.

For debian also fixes are moving along after i reached our to debian-lts list ass Christoph suggested earlier.

> reconfirm our tests. For example, this appears to also break RedHat 6

But good news is also impact is getting smaller with less distro/releases affected as some are getting fixed.

Also with few combinations being affected maybe just some extra note/section in the wiki could be enough.
wget http:// for affected + extra step to verify key fingerprint

For other distros (RHEL,Centos,Suse, ...) do we have anyone having contact's there nearby here?
I think there was yum.postgresql.org with Devrim active on that (but don't remember).
As probably reaching out to them maybe they are also interested in just backporting the libary fixes on their side.

Stefan