public inbox for [email protected]
help / color / mirror / Atom feedservice outage on panora.postgresql.org
4+ messages / 1 participants
[nested] [flat]
* service outage on panora.postgresql.org
@ 2021-11-13 21:46 Stefan Kaltenbrunner <[email protected]>
2021-11-13 22:29 ` Re: service outage on panora.postgresql.org Stefan Kaltenbrunner <[email protected]>
0 siblings, 1 reply; 4+ messages in thread
From: Stefan Kaltenbrunner @ 2021-11-13 21:46 UTC (permalink / raw)
To: PostgreSQL WWW <[email protected]>
Hi all!
We are currently experiencing an outage of one of our
vmhosts(panora.postgresql.org) which is affecting the following
production services(among some internal systems):
brekka.postgresql.org (aka buildfarm.postgresql.org)
We are working on the issue but atm I have not ETA for a solution
because it is not entirely clear what the actual issue is (other than
that the box has no working network atm).
Stefan
^ permalink raw reply [nested|flat] 4+ messages in thread
* Re: service outage on panora.postgresql.org
2021-11-13 21:46 service outage on panora.postgresql.org Stefan Kaltenbrunner <[email protected]>
@ 2021-11-13 22:29 ` Stefan Kaltenbrunner <[email protected]>
2021-11-14 08:52 ` Re: service outage on panora.postgresql.org Stefan Kaltenbrunner <[email protected]>
0 siblings, 1 reply; 4+ messages in thread
From: Stefan Kaltenbrunner @ 2021-11-13 22:29 UTC (permalink / raw)
To: PostgreSQL WWW <[email protected]>
On 11/13/21 10:46 PM, Stefan Kaltenbrunner wrote:
> Hi all!
>
> We are currently experiencing an outage of one of our
> vmhosts(panora.postgresql.org) which is affecting the following
> production services(among some internal systems):
>
> brekka.postgresql.org (aka buildfarm.postgresql.org)
>
>
> We are working on the issue but atm I have not ETA for a solution
> because it is not entirely clear what the actual issue is (other than
> that the box has no working network atm).
to be more specific on this - those systems lost IPv4 connectivity but
are still reachable over IPv6 but also all services are impaired by
being unable to connect to any IPv4 host(inkluding DNS).
Stefan
^ permalink raw reply [nested|flat] 4+ messages in thread
* Re: service outage on panora.postgresql.org
2021-11-13 21:46 service outage on panora.postgresql.org Stefan Kaltenbrunner <[email protected]>
2021-11-13 22:29 ` Re: service outage on panora.postgresql.org Stefan Kaltenbrunner <[email protected]>
@ 2021-11-14 08:52 ` Stefan Kaltenbrunner <[email protected]>
2021-11-15 19:27 ` Re: service outage on panora.postgresql.org Stefan Kaltenbrunner <[email protected]>
0 siblings, 1 reply; 4+ messages in thread
From: Stefan Kaltenbrunner @ 2021-11-14 08:52 UTC (permalink / raw)
To: PostgreSQL WWW <[email protected]>
On 11/13/21 11:29 PM, Stefan Kaltenbrunner wrote:
> On 11/13/21 10:46 PM, Stefan Kaltenbrunner wrote:
>> Hi all!
>>
>> We are currently experiencing an outage of one of our
>> vmhosts(panora.postgresql.org) which is affecting the following
>> production services(among some internal systems):
>>
>> brekka.postgresql.org (aka buildfarm.postgresql.org)
>>
>>
>> We are working on the issue but atm I have not ETA for a solution
>> because it is not entirely clear what the actual issue is (other than
>> that the box has no working network atm).
>
> to be more specific on this - those systems lost IPv4 connectivity but
> are still reachable over IPv6 but also all services are impaired by
> being unable to connect to any IPv4 host(inkluding DNS).
services should be back for now - root cause is still somewhat unclear
and we will likely need a few more reboots of the box in the next few
days to nail this down...
Sorry for the inconvenience :/
Stefan
^ permalink raw reply [nested|flat] 4+ messages in thread
* Re: service outage on panora.postgresql.org
2021-11-13 21:46 service outage on panora.postgresql.org Stefan Kaltenbrunner <[email protected]>
2021-11-13 22:29 ` Re: service outage on panora.postgresql.org Stefan Kaltenbrunner <[email protected]>
2021-11-14 08:52 ` Re: service outage on panora.postgresql.org Stefan Kaltenbrunner <[email protected]>
@ 2021-11-15 19:27 ` Stefan Kaltenbrunner <[email protected]>
0 siblings, 0 replies; 4+ messages in thread
From: Stefan Kaltenbrunner @ 2021-11-15 19:27 UTC (permalink / raw)
To: PostgreSQL WWW <[email protected]>
On 11/14/21 9:52 AM, Stefan Kaltenbrunner wrote:
> On 11/13/21 11:29 PM, Stefan Kaltenbrunner wrote:
>> On 11/13/21 10:46 PM, Stefan Kaltenbrunner wrote:
>>> Hi all!
>>>
>>> We are currently experiencing an outage of one of our
>>> vmhosts(panora.postgresql.org) which is affecting the following
>>> production services(among some internal systems):
>>>
>>> brekka.postgresql.org (aka buildfarm.postgresql.org)
>>>
>>>
>>> We are working on the issue but atm I have not ETA for a solution
>>> because it is not entirely clear what the actual issue is (other than
>>> that the box has no working network atm).
>>
>> to be more specific on this - those systems lost IPv4 connectivity but
>> are still reachable over IPv6 but also all services are impaired by
>> being unable to connect to any IPv4 host(inkluding DNS).
>
> services should be back for now - root cause is still somewhat unclear
> and we will likely need a few more reboots of the box in the next few
> days to nail this down...
>
>
> Sorry for the inconvenience :/
we have found the root-cause of this issue and it is caused by
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=949062
We have locally backported the fixes to the affected package and the
service should now be stable again.
Also special thanks to the great support team from equinix metal who
helped in diagnosing the issue!
Stefan
^ permalink raw reply [nested|flat] 4+ messages in thread
end of thread, other threads:[~2021-11-15 19:27 UTC | newest]
Thread overview: 4+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2021-11-13 21:46 service outage on panora.postgresql.org Stefan Kaltenbrunner <[email protected]>
2021-11-13 22:29 ` Stefan Kaltenbrunner <[email protected]>
2021-11-14 08:52 ` Stefan Kaltenbrunner <[email protected]>
2021-11-15 19:27 ` Stefan Kaltenbrunner <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox