Date: Tue, 9 Jun 2020 07:09:59 -0500
From: Justin Pryzby <pryzby@telsasoft.com>
To: Krzysztof Olszewski <kolszew73@gmail.com>
Cc: Pavel Stehule <pavel.stehule@gmail.com>,
	Pgsql Performance <pgsql-performance@lists.postgresql.org>
Subject: Re: Postgresql server gets stuck at low load
Message-ID: <20200609120959.GC22473@telsasoft.com>
References: 
 <CAHihO3wAmb_b=uMT86PhJYk3-C-4_pNBqJ-TUehZVk82ifFFDw@mail.gmail.com>
 <CAFj8pRDERh_L7qHzRyG4QG4dRTfYZBcYOOTO0-0nTRBHTK+PVA@mail.gmail.com>
 <CAHihO3yZ2698bBgt2EMocMhy=SRAUMjw7KZ2u_36EaKB4Mcp_Q@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: 
 <CAHihO3yZ2698bBgt2EMocMhy=SRAUMjw7KZ2u_36EaKB4Mcp_Q@mail.gmail.com>
User-Agent: Mutt/1.9.4 (2018-02-28)
Precedence: bulk

On Tue, Jun 09, 2020 at 01:54:21PM +0200, Krzysztof Olszewski wrote:
>  I had hugepage's off and on, problems still occurs,
> thanx for "perf top" suggestion,

> pt., 5 cze 2020 o 13:38 Pavel Stehule <pavel.stehule@gmail.com> napisał(a):
> > pá 5. 6. 2020 v 12:07 odesílatel Krzysztof Olszewski <kolszew73@gmail.com> napsal:
> >
> >> I have problem with one of my Postgres production server. Server works
> >> fine almost always, but sometimes without any increase of transactions or
> >> statements amount, machine gets stuck. Cores goes up to 100%, load up to
> >> 160%. When it happens then there are problems with connect to database and
> >> even it will succeed, simple queries works several seconds instead of
> >> milliseconds.Problem sometimes stops after a period a time (e.g. 35 min),
> >> sometimes we must restart Postgres, Linux, or even KVM (which exists as
> >> virtualization host).
> >>
> >> My hardware
> >> 56 cores (Intel Core Processor (Skylake, IBRS))
> >> 400 GB RAM
> >> RAID10 with about 40k IOPS
> >>
> >> shared_buffers 64000 MB
> >>
> >> In correlation with stucks i see informations in kernel log about
> >> NMI watchdog: BUG: soft lockup - CPU#25 stuck for 23s! [postmaster:33935]
> >
> > https://www.postgresql.org/message-id/CAHyXU0yAsVxoab2PcyoCuPjqymtnaE93v7bN4ctv2aNi92fefA%40mail.gmail.com
> >
> > Can be interesting a reply on Merlin's question from mail/.
> >
> > cat /sys/kernel/mm/redhat_transparent_hugepage/enabled
> > cat /sys/kernel/mm/redhat_transparent_hugepage/defrag

try this:
echo 2 |sudo /sys/kernel/mm/ksm/run

https://www.postgresql.org/message-id/20170718180152.GE17566%40telsasoft.com

-- 
Justin