Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1rnxsA-00GFpf-8T for pgsql-general@arkaria.postgresql.org; Sat, 23 Mar 2024 09:41:06 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.94.2) (envelope-from ) id 1rnxs7-009Usq-RG for pgsql-general@arkaria.postgresql.org; Sat, 23 Mar 2024 09:41:04 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1rnxs7-009Usi-Gd for pgsql-general@lists.postgresql.org; Sat, 23 Mar 2024 09:41:03 +0000 Received: from ms-10.1blu.de ([178.254.4.101]) by magus.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1rnxs0-0063gk-GX for pgsql-general@lists.postgresql.org; Sat, 23 Mar 2024 09:41:03 +0000 Received: from [62.216.210.157] (helo=localhost.unixarea.de) by ms-10.1blu.de with esmtpsa (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1rnxrz-005PLW-OH; Sat, 23 Mar 2024 10:40:55 +0100 Received: from localhost.my.domain (c720-1400094 [127.0.0.1]) by localhost.unixarea.de (8.17.1/8.14.9) with ESMTP id 42N9etXv001577; Sat, 23 Mar 2024 10:40:55 +0100 (CET) (envelope-from guru@unixarea.de) Received: (from guru@localhost) by localhost.my.domain (8.17.1/8.14.9/Submit) id 42N9esbJ001576; Sat, 23 Mar 2024 10:40:54 +0100 (CET) (envelope-from guru@unixarea.de) X-Authentication-Warning: localhost.my.domain: guru set sender to guru@unixarea.de using -f Date: Sat, 23 Mar 2024 10:40:54 +0100 From: Matthias Apitz To: Ron Johnson Cc: "pgsql-generallists.postgresql.org" Subject: Re: soft lockup - CPU#16 stuck for 3124s! [postmaster:2273] Message-ID: Reply-To: Matthias Apitz References: <944273.1711128435@sss.pgh.pa.us> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-Operating-System: FreeBSD 14.0-CURRENT r1400094 (amd64) X-message-flag: Mails in HTML will not be read! Please send only plain text. X-Con-Id: 51246 X-Con-U: 0-guru X-Originating-IP: 62.216.210.157 List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk El día viernes, marzo 22, 2024 a las 01:31:43p. m. -0400, Ron Johnson escribió: > On Fri, Mar 22, 2024 at 1:27 PM Tom Lane wrote: > > > Matthias Apitz writes: > > > We have a PostgreSQL 15.1 server in production at a customer for some > > > weeks (migrated from an older version) on SuSE SLES 15. > > > > > The customer is facing machine locks and before the Linux server does > > > not respond any more (not even on SSH, only power-cycle reset helps to > > > get it back), short before the fault a lot of messages are in > > > /var/log/messages of the content: > > > > > # grep watchdog: /var/log/messages > > > ... > > > 2024-03-22T13:11:32.056154+01:00 sunrise kernel: [327844.313048][ C25] > > watchdog: BUG: soft lockup - CPU#25 stuck for 3069s! [migration/25:166] > > > 2024-03-22T13:12:28.056244+01:00 sunrise kernel: [327900.310267][ C16] > > watchdog: BUG: soft lockup - CPU#16 stuck for 3124s! [postmaster:2273] > > > 2024-03-22T13:12:28.056340+01:00 sunrise kernel: [327900.311052][ C25] > > watchdog: BUG: soft lockup - CPU#25 stuck for 3121s! [migration/25:166] > > > > Sounds like failing hardware to me :-( > > > Updating to 15.6 would rule out any bugs squashed in the last 15 months. Yesterday the message appeared 300 times: sunrise:~ # xz -dc /var/log/messages-20240323.xz | grep lockup | wc -l 323 Since the WAL copy and backup is working correctly, no such message anymore. matthias -- Matthias Apitz, ✉ guru@unixarea.de, http://www.unixarea.de/ +49-176-38902045 Public GnuPG key: http://www.unixarea.de/key.pub