Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1jO0Ih-0007lo-Dz for pgsql-performance@arkaria.postgresql.org; Mon, 13 Apr 2020 14:43:03 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1jO0Ig-0000U2-AD for pgsql-performance@arkaria.postgresql.org; Mon, 13 Apr 2020 14:43:02 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1jO0If-0000Tv-Vg for pgsql-performance@lists.postgresql.org; Mon, 13 Apr 2020 14:43:02 +0000 Received: from mail-qk1-x734.google.com ([2607:f8b0:4864:20::734]) by magus.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.92) (envelope-from ) id 1jO0Id-0006Qn-5V for pgsql-performance@lists.postgresql.org; Mon, 13 Apr 2020 14:43:01 +0000 Received: by mail-qk1-x734.google.com with SMTP id v7so9666464qkc.0 for ; Mon, 13 Apr 2020 07:42:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=telsasoft-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=9OnOc8bb0dfNz85Lm8f4xdkTR901KI9vNWiO5sSIzrs=; b=IagyITtVIoUolqL7Ox4ROhuFaOieoyQGUHUUV5Ouurycdr0TypRHBtkgmNjuv4tCkb dOQBw09F4G7Lyy6aBEgipXaDdpvolKO1gcM40yXsfK1TwNc5axSfc7NRbCQs3XUpRkXh hVVHoKl9JtrEA4DMYricoWDLS3SCxs0dUKFecZOF7KN+3jG4C77seU3tJYvXjOSVyPe1 Sb6ywc35pAe7cBhy0MXoKqBjwPJ40Gj8r4tGVu+g4387Az1DT7QyvVyV3kBuzBYPVE9/ wl6ZrniaBjreQHMmzC7IcBjwBZNJTDxlNcdTr2HLX6TVl3j8HpLCtSnKP616OtV+y9f2 OQzw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to:user-agent; bh=9OnOc8bb0dfNz85Lm8f4xdkTR901KI9vNWiO5sSIzrs=; b=F87cz+0avq3+58smw62fVQ13DpJX6O2yKEKrt8IjBD+HC6GIT8NTPF2TSDzGvFLtqX Ud0qBIS2cetX/jdXrBpa7HzTB0VJALSUoI7ZDgNgov8x2ATHQQNgn62lgwpKnPBlVEWB /O/QO83VtyOYsynoMsKuCsaC1uAq66d/XUIaZO0hZa6h8kWrCsttRIDMqNbevx/LqUTE Iaw2KnT8GC1kK+hzaRu1RxTztdvZ0Z+z2/wSTsVB31mgH9gR0N6L7jMYneFMhuGShf4n 8Yy3LETAFPqYasCUwfnsJ21iGrqzLbNeu5xgALBoaZs3u/hUQOdET1JX0I+XJU/tzZaz Hw9Q== X-Gm-Message-State: AGi0PuZmYS9G7lGwtcLFC61gugdpzVJSmTutT4RQoxR07SYdZKnpUi8T i2vRr3gLm92eNbhb39CDPzU540KR5zE= X-Google-Smtp-Source: APiQypLGzfSgFhtgYGRkdeI5BJYlXa/Mfl/YuigVfa3I88gUswGL/fWPA4lXOgDT6cXNTCavyKzGcA== X-Received: by 2002:a37:88d:: with SMTP id 135mr16230254qki.211.1586788976948; Mon, 13 Apr 2020 07:42:56 -0700 (PDT) Received: from pryzbyj (charmander.telsasoft.com. [50.244.222.1]) by smtp.gmail.com with ESMTPSA id e9sm8075316qtq.82.2020.04.13.07.42.55 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Mon, 13 Apr 2020 07:42:56 -0700 (PDT) Received: by pryzbyj (Postfix, from userid 1000) id 9AA66800897; Mon, 13 Apr 2020 09:42:54 -0500 (CDT) Date: Mon, 13 Apr 2020 09:42:54 -0500 From: Justin Pryzby To: Don Seiler Cc: pgsql-performance@lists.postgresql.org Subject: Re: High kswapd Message-ID: <20200413144254.GS2228@telsasoft.com> References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.9.4 (2018-02-28) List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Precedence: bulk On Mon, Apr 13, 2020 at 09:34:23AM -0500, Don Seiler wrote: > Good morning, > > Ubuntu 16.04.6 LTS > PostgreSQL 9.6.5 > > On one of our database servers, we're regularly seeing kswapd at the top of > "top" output, regularly using over 50 %CPU. We should have well over 80GB > of available memory according to "free -m". Do you have THP enabled? Or KSM ? tail /sys/kernel/mm/ksm/run /sys/kernel/mm/transparent_hugepage/khugepaged/defrag /sys/kernel/mm/transparent_hugepage/enabled /sys/kernel/mm/transparent_hugepage/defrag https://www.postgresql.org/message-id/20170718180152.GE17566%40telsasoft.com -- Justin