pg.ddx.io pgsql-performance@postgresql.org mailing list archive
help / color / mirror / Atom feedHigh kswapd
5+ messages / 2 participants
[nested] [flat]
* High kswapd
@ 2020-04-13 14:34 Don Seiler <don@seiler.us>
0 siblings, 1 reply; 5+ messages in thread
From: Don Seiler @ 2020-04-13 14:34 UTC (permalink / raw)
To: pgsql-performance@lists.postgresql.org
Good morning,
Ubuntu 16.04.6 LTS
PostgreSQL 9.6.5
On one of our database servers, we're regularly seeing kswapd at the top of
"top" output, regularly using over 50 %CPU. We should have well over 80GB
of available memory according to "free -m".
# free -m
total used free shared buff/cache
available
Mem: 125910 41654 820 857 83435
82231
Swap: 511 448 63
We've already got vm.swappiness and vm.zone_reclaim_mode set to 0, and NUMA
is disabled from what I can see:
# dmesg | grep -i numa
[ 0.000000] No NUMA configuration found
We are using HugePages and things look good there as well. Curious what
would be causing kswapd to run hot like it is, or if it is a red herring as
I look into high CPU usage on this box (although, again, it is the
single-highest CPU user).
--
Don Seiler
www.seiler.us
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: High kswapd
@ 2020-04-13 14:42 Justin Pryzby <pryzby@telsasoft.com>
parent: Don Seiler <don@seiler.us>
0 siblings, 1 reply; 5+ messages in thread
From: Justin Pryzby @ 2020-04-13 14:42 UTC (permalink / raw)
To: Don Seiler <don@seiler.us>; +Cc: pgsql-performance@lists.postgresql.org
On Mon, Apr 13, 2020 at 09:34:23AM -0500, Don Seiler wrote:
> Good morning,
>
> Ubuntu 16.04.6 LTS
> PostgreSQL 9.6.5
>
> On one of our database servers, we're regularly seeing kswapd at the top of
> "top" output, regularly using over 50 %CPU. We should have well over 80GB
> of available memory according to "free -m".
Do you have THP enabled? Or KSM ?
tail /sys/kernel/mm/ksm/run /sys/kernel/mm/transparent_hugepage/khugepaged/defrag /sys/kernel/mm/transparent_hugepage/enabled /sys/kernel/mm/transparent_hugepage/defrag
https://www.postgresql.org/message-id/20170718180152.GE17566%40telsasoft.com
--
Justin
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: High kswapd
@ 2020-04-13 14:46 Don Seiler <don@seiler.us>
parent: Justin Pryzby <pryzby@telsasoft.com>
0 siblings, 1 reply; 5+ messages in thread
From: Don Seiler @ 2020-04-13 14:46 UTC (permalink / raw)
To: Justin Pryzby <pryzby@telsasoft.com>; +Cc: pgsql-performance@lists.postgresql.org
On Mon, Apr 13, 2020 at 9:42 AM Justin Pryzby <pryzby@telsasoft.com> wrote:
>
> tail /sys/kernel/mm/ksm/run
> /sys/kernel/mm/transparent_hugepage/khugepaged/defrag
> /sys/kernel/mm/transparent_hugepage/enabled
> /sys/kernel/mm/transparent_hugepage/defrag
>
# tail /sys/kernel/mm/ksm/run
/sys/kernel/mm/transparent_hugepage/khugepaged/defrag
/sys/kernel/mm/transparent_hugepage/enabled
/sys/kernel/mm/transparent_hugepage/defrag
==> /sys/kernel/mm/ksm/run <==
0
==> /sys/kernel/mm/transparent_hugepage/khugepaged/defrag <==
1
==> /sys/kernel/mm/transparent_hugepage/enabled <==
always madvise [never]
==> /sys/kernel/mm/transparent_hugepage/defrag <==
always madvise [never]
--
Don Seiler
www.seiler.us
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: High kswapd
@ 2020-04-13 14:58 Justin Pryzby <pryzby@telsasoft.com>
parent: Don Seiler <don@seiler.us>
0 siblings, 1 reply; 5+ messages in thread
From: Justin Pryzby @ 2020-04-13 14:58 UTC (permalink / raw)
To: Don Seiler <don@seiler.us>; +Cc: pgsql-performance@lists.postgresql.org
On Mon, Apr 13, 2020 at 09:46:22AM -0500, Don Seiler wrote:
> ==> /sys/kernel/mm/ksm/run <==
> 0
Was it off to begin with ?
If not, you can set it to "2" to "unshare" pages.
> ==> /sys/kernel/mm/transparent_hugepage/khugepaged/defrag <==
> 1
So I'd suggest trying with this disabled.
I don't know if I ever fully understood the problem, but it sounds like at
least in your case it's related to large shared_buffers, and hugepages, which
cannot be swapped out.
--
Justin
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: High kswapd
@ 2020-04-13 15:08 Don Seiler <don@seiler.us>
parent: Justin Pryzby <pryzby@telsasoft.com>
0 siblings, 0 replies; 5+ messages in thread
From: Don Seiler @ 2020-04-13 15:08 UTC (permalink / raw)
To: Justin Pryzby <pryzby@telsasoft.com>; +Cc: pgsql-performance@lists.postgresql.org
On Mon, Apr 13, 2020 at 9:58 AM Justin Pryzby <pryzby@telsasoft.com> wrote:
> On Mon, Apr 13, 2020 at 09:46:22AM -0500, Don Seiler wrote:
> > ==> /sys/kernel/mm/ksm/run <==
> > 0
>
> Was it off to begin with ?
> If not, you can set it to "2" to "unshare" pages.
>
Yes we haven't changed this. It was already set to 0.
>
> > ==> /sys/kernel/mm/transparent_hugepage/khugepaged/defrag <==
> > 1
>
> So I'd suggest trying with this disabled.
>
My understanding was that THP is disabled anyway. What would this defrag
feature be doing now?
> I don't know if I ever fully understood the problem, but it sounds like at
> least in your case it's related to large shared_buffers, and hugepages,
> which
> cannot be swapped out.
>
Basically the problem is our DB host getting slammed with connections (even
with pgbouncer in place). We see the CPU load spiking, and when we check
"top" we regularly see "kswapd" at the top of the list. For example, just
now kswapd is at 72 %CPU in top. The next highest is a postgres process at
6.6 %CPU.
Our shared_buffers is set to 32GB, and HugePages is set to 36GB:
# grep Huge /proc/meminfo
AnonHugePages: 0 kB
HugePages_Total: 18000
HugePages_Free: 1897
HugePages_Rsvd: 41
HugePages_Surp: 0
Hugepagesize: 2048 kB
Also FWIW this host is actually a VSphere VM. We're looking into any
underlying events during these spikes as well.
Don.
--
Don Seiler
www.seiler.us
^ permalink raw reply [nested|flat] 5+ messages in thread
end of thread, other threads:[~2020-04-13 15:08 UTC | newest]
Thread overview: 5+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2020-04-13 14:34 High kswapd Don Seiler <don@seiler.us>
2020-04-13 14:42 ` Justin Pryzby <pryzby@telsasoft.com>
2020-04-13 14:46 ` Don Seiler <don@seiler.us>
2020-04-13 14:58 ` Justin Pryzby <pryzby@telsasoft.com>
2020-04-13 15:08 ` Don Seiler <don@seiler.us>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox