From: Matthias Apitz <guru@unixarea.de>
To: Ron Johnson <ronljohnsonjr@gmail.com>
Cc: pgsql-generallists.postgresql.org <pgsql-general@lists.postgresql.org>
Subject: Re: soft lockup - CPU#16 stuck for 3124s! [postmaster:2273]
Date: Sat, 23 Mar 2024 10:40:54 +0100
Message-ID: <Zf6jpo0Zb__PrxIk@c720-1400094> (raw)
In-Reply-To: <CANzqJaAda267=Noy_bQceGdvHTVs+fQm=AixhRdiqe6rWfZ07g@mail.gmail.com>
References: <Zf27/PpLpsE0fB+i@pureos>
<944273.1711128435@sss.pgh.pa.us>
<CANzqJaAda267=Noy_bQceGdvHTVs+fQm=AixhRdiqe6rWfZ07g@mail.gmail.com>
El día viernes, marzo 22, 2024 a las 01:31:43p. m. -0400, Ron Johnson escribió:
> On Fri, Mar 22, 2024 at 1:27 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
>
> > Matthias Apitz <guru@unixarea.de> writes:
> > > We have a PostgreSQL 15.1 server in production at a customer for some
> > > weeks (migrated from an older version) on SuSE SLES 15.
> >
> > > The customer is facing machine locks and before the Linux server does
> > > not respond any more (not even on SSH, only power-cycle reset helps to
> > > get it back), short before the fault a lot of messages are in
> > > /var/log/messages of the content:
> >
> > > # grep watchdog: /var/log/messages
> > > ...
> > > 2024-03-22T13:11:32.056154+01:00 sunrise kernel: [327844.313048][ C25]
> > watchdog: BUG: soft lockup - CPU#25 stuck for 3069s! [migration/25:166]
> > > 2024-03-22T13:12:28.056244+01:00 sunrise kernel: [327900.310267][ C16]
> > watchdog: BUG: soft lockup - CPU#16 stuck for 3124s! [postmaster:2273]
> > > 2024-03-22T13:12:28.056340+01:00 sunrise kernel: [327900.311052][ C25]
> > watchdog: BUG: soft lockup - CPU#25 stuck for 3121s! [migration/25:166]
> >
> > Sounds like failing hardware to me :-(
>
>
> Updating to 15.6 would rule out any bugs squashed in the last 15 months.
Yesterday the message appeared 300 times:
sunrise:~ # xz -dc /var/log/messages-20240323.xz | grep lockup | wc -l
323
Since the WAL copy and backup is working correctly, no such message
anymore.
matthias
--
Matthias Apitz, ✉ guru@unixarea.de, http://www.unixarea.de/ +49-176-38902045
Public GnuPG key: http://www.unixarea.de/key.pub
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients from your mail client; the link fills in
the headers below: reply via email
Some clients, webmail especially, drop In-Reply-To and References from
a link, and the reply then starts a new thread: use the mbox below.
To: pgsql-general@postgresql.org
Cc: guru@unixarea.de, ronljohnsonjr@gmail.com
Subject: Re: soft lockup - CPU#16 stuck for 3124s! [postmaster:2273]
In-Reply-To: <Zf6jpo0Zb__PrxIk@c720-1400094>
References: <Zf27/PpLpsE0fB+i@pureos> <944273.1711128435@sss.pgh.pa.us> <CANzqJaAda267=Noy_bQceGdvHTVs+fQm=AixhRdiqe6rWfZ07g@mail.gmail.com> <Zf6jpo0Zb__PrxIk@c720-1400094>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox