Subject: service outage on panora.postgresql.org

From: Stefan Kaltenbrunner <[email-masked]>
Date: Sat, 13 Nov 2021 22:46:48 +0100

Hi all!

We are currently experiencing an outage of one of our 
vmhosts(panora.postgresql.org) which is affecting the following 
production services(among some internal systems):

brekka.postgresql.org (aka buildfarm.postgresql.org)

We are working on the issue but atm I have not ETA for a solution 
because it is not entirely clear what the actual issue is (other than 
that the box has no working network atm).

Stefan

---

From: Stefan Kaltenbrunner <[email-masked]>
Date: Sat, 13 Nov 2021 23:29:03 +0100

> that the box has no working network atm).

to be more specific on this - those systems lost IPv4 connectivity but 
are still reachable over IPv6 but also all services are impaired by 
being unable to connect to any IPv4 host(inkluding DNS).

Stefan

---

From: Stefan Kaltenbrunner <[email-masked]>
Date: Sun, 14 Nov 2021 09:52:36 +0100

> being unable to connect to any IPv4 host(inkluding DNS).

services should be back for now - root cause is still somewhat unclear 
and we will likely need a few more reboots of the box in the next few 
days to nail this down...

Sorry for the inconvenience :/

Stefan

---

From: Stefan Kaltenbrunner <[email-masked]>
Date: Mon, 15 Nov 2021 20:27:01 +0100

> Sorry for the inconvenience :/

we have found the root-cause of this issue and it is caused by 
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=949062

We have locally backported the fixes to the affected package and the 
service should now be stable again.

Also special thanks to the great support team from equinix metal who 
helped in diagnosing the issue!

Stefan