Date: Mon, 20 Jul 2020 18:08:47 -0400
From: Alvaro Herrera <alvherre@2ndquadrant.com>
To: jian xu <jamesxu@outlook.com>
Cc: MichaelDBA <MichaelDBA@sqlexec.com>,
	Scott Ribe <scott_ribe@elevated-dev.com>,
	Laurenz Albe <laurenz.albe@cybertec.at>,
	"pgsql-admin@postgresql.org" <pgsql-admin@postgresql.org>
Subject: Re: checkpoint process use too much memory
Message-ID: <20200720220847.GA28454@alvherre.pgsql>
MIME-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: 
 <DM6PR05MB68270B6144BDF688D3849436A17B0@DM6PR05MB6827.namprd05.prod.outlook.com>
User-Agent: Mutt/1.10.1 (2018-07-13)
Precedence: bulk

On 2020-Jul-20, jian xu wrote:

>       we are seeing this issue again, the checkpoint process is eating up all the memory,
> based on the htop, the checkpointer process has
> VIRT: 103G
> RES: 103G
> SHR: 4099M
> MEM%: 41%
> 
> I use ps_mem.py to check the process,
> https://github.com/pixelb/ps_mem
> =================================
>  Private  +   Shared  =  RAM used Program
> 
>  99.5 GiB +  37.5 MiB =  99.5 GiB postgres
> ---------------------------------
>                                            99.5 GiB
> =================================
> my pg version is 10.3 , shared_buffer size is 4GB. work_mem 64MB. there are about 120 connections on the db side. most of them are idle.
> does anyone have idea how to troubleshooting this kind of issue?

I see no reason for checkpointer to use that much memory.  Accounting
for the whole of shared_buffers is okay (as Laurenz explaijned), which
in other installations may account for several gigabytes of memory.  But
if shared_buffers is 4GB, then 99GB is clearly overboard.  I see no
explanation for checkpointer using 25x shared_buffers.

Maybe looking at /proc/<PID>/smaps would tell you where that memory is
going?  Have a look there are see if you can figure out some obviously
absurd entries.

-- 
Álvaro Herrera                https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services