Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1mmhdH-0008Mj-Sy for pgsql-www@arkaria.postgresql.org; Mon, 15 Nov 2021 19:27:12 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1mmhdG-000090-Rs for pgsql-www@arkaria.postgresql.org; Mon, 15 Nov 2021 19:27:10 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1mmhdG-000081-KZ for pgsql-www@lists.postgresql.org; Mon, 15 Nov 2021 19:27:10 +0000 Received: from imp.madness.at ([2a02:16a8:dc41::218]) by makus.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1mmhdD-0005GS-OH for pgsql-www@lists.postgresql.org; Mon, 15 Nov 2021 19:27:09 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=kaltenbrunner.cc; s=20190215; h=Content-Transfer-Encoding:Content-Type: In-Reply-To:MIME-Version:Date:Message-ID:References:To:From:Subject:Sender: Reply-To:Cc:Content-ID:Content-Description; bh=psiGRxCMBeTQvos+zntIEQRw0BZ0hkgqAAzpgRktA1U=; b=bKLnEkYb75AahiX/84XNRPUhax uuGIWmKDBWosMzJQ4C1S+xz6hgPH6Es4cAToZFFsb+n/YfAa5SlumYw/cYcitfFZFD0rH0ZPBnAo5 33vMOWFM58f6hse2BpuILEEMj2FBhKDdn46TNwEQmTVNmb8pUCUy6yCiD0uXx8PhibMXHXyGR1fm8 0v8gWdx/bsO+VFRHMBb3MRMT/xHb4kdVOlOF86k2W0bPhjo22coESQmgebw7eB02TqfEbs8Um96Fo nim3QkPkaribI/9N+bxw8J0WcicacTwSYcTmfT5Xl3Pko/1TLlJonmNMnutYUh5Ue3CxZjc65TvEX N+wF2Sng==; Received: from [83.215.245.171] (helo=[192.168.8.98]) by imp.madness.at with esmtpsa (TLS1.3:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.92) (envelope-from ) id 1mmhd8-0005cN-UV for pgsql-www@lists.postgresql.org; Mon, 15 Nov 2021 20:27:05 +0100 Subject: Re: service outage on panora.postgresql.org From: Stefan Kaltenbrunner To: PostgreSQL WWW References: <4eba3256-fbb0-8a44-21cb-9566b8a564b0@kaltenbrunner.cc> Message-ID: <021f15b1-bc3e-af14-9d71-fdf52c8df11b@kaltenbrunner.cc> Date: Mon, 15 Nov 2021 20:27:01 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.14.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk On 11/14/21 9:52 AM, Stefan Kaltenbrunner wrote: > On 11/13/21 11:29 PM, Stefan Kaltenbrunner wrote: >> On 11/13/21 10:46 PM, Stefan Kaltenbrunner wrote: >>> Hi all! >>> >>> We are currently experiencing an outage of one of our >>> vmhosts(panora.postgresql.org) which is affecting the following >>> production services(among some internal systems): >>> >>> brekka.postgresql.org (aka buildfarm.postgresql.org) >>> >>> >>> We are working on the issue but atm I have not ETA for a solution >>> because it is not entirely clear what the actual issue is (other than >>> that the box has no working network atm). >> >> to be more specific on this - those systems lost IPv4 connectivity but >> are still reachable over IPv6 but also all services are impaired by >> being unable to connect to any IPv4 host(inkluding DNS). > > services should be back for now - root cause is still somewhat unclear > and we will likely need a few more reboots of the box in the next few > days to nail this down... > > > Sorry for the inconvenience :/ we have found the root-cause of this issue and it is caused by https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=949062 We have locally backported the fixes to the affected package and the service should now be stable again. Also special thanks to the great support team from equinix metal who helped in diagnosing the issue! Stefan