Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1u1PFO-001v5B-ET for pgsql-performance@arkaria.postgresql.org; Sun, 06 Apr 2025 12:37:10 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.94.2) (envelope-from ) id 1u1PFL-00DBE1-87 for pgsql-performance@arkaria.postgresql.org; Sun, 06 Apr 2025 12:37:07 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1u1PFK-00DBDt-PQ for pgsql-performance@lists.postgresql.org; Sun, 06 Apr 2025 12:37:07 +0000 Received: from mail-pg1-x535.google.com ([2607:f8b0:4864:20::535]) by magus.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.96) (envelope-from ) id 1u1PFI-003il6-2P for pgsql-performance@postgresql.org; Sun, 06 Apr 2025 12:37:06 +0000 Received: by mail-pg1-x535.google.com with SMTP id 41be03b00d2f7-7fd581c2bf4so2651036a12.3 for ; Sun, 06 Apr 2025 05:37:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1743943022; x=1744547822; darn=postgresql.org; h=to:subject:message-id:date:from:mime-version:from:to:cc:subject :date:message-id:reply-to; bh=rjlFpRV9l3bjusUpqWQRkofD2dTUbNwHQ9LCbS9FZnU=; b=VtLW5plg+OYPt2Ve5Yezv1pt05R4Ob7sL6JbPbQ7v0KjbPbkE1ey/YFL69B1FJ8+eg emW/owBPpXzYXtn0vcmXBeILx3J7UfxK8QRo49iEpDuLjK5gQlRmvMe2x8M5QSQ4iCQV iXTUYmv+9yLt4BYZah7qTQ9UwecpGVlSrqdwkax564Jee7iMZNbZRwpsehTJU5dEJYs6 0tKgpm1VzAf+jpjsgoQeJ0MZ3BwOsod2lNsPT97nNZLdZIYqavIVENf1JxNLJMSZH40o bi8HkJAdoRfgVHZdeRBCYyLkUtVTztpXSAaB+kySP4UhDWZ2k6yiHby0fkLUmADSVwNF DbQw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1743943022; x=1744547822; h=to:subject:message-id:date:from:mime-version:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=rjlFpRV9l3bjusUpqWQRkofD2dTUbNwHQ9LCbS9FZnU=; b=T2m/rrGPvuJ47EGdqBJiqwPpaXFRFZU1nwVjDmUBihIk9au/TRFEvZq+kr3CZhQv6H 1TuR9CPuQ804q2txWUfQNae/gwVwxFkId2IEbq72QMz1Hs0JjrW68XVTy5vJqkkS1RjG OqfixekPreT8GBDf9+VNjyZnrcq65WoZkVtZ92WKbRYaxcJ0jR5cOKnAwHo6U03sSn6N iCFr0vhxQZsex3eq6x95LBcwpv6/1dZFQ+wKbixqpJT4/OjbAkrEi9VzTJni315AG3GI 2iiekeKDt5XTY622hJI3mnGV79aZP1L3ekI0IMegPoYJE6N9dA2orXQlgAe9jKngVWyc qlqw== X-Gm-Message-State: AOJu0Ywa5bTVB1PMV6ILrDbVs3gGUfYMPYXd7vVnYX80hVpeG6HH4GKK yp6Ij2QOTKwD+BQ5K7UR3A/02VmH1Orc9CzBWzegyOo71TNjakVWXzur9RddDN+AtU1qRzB2TpV 6JUx6txIc8qGMHvUfyVMnt3TjhK0g1g== X-Gm-Gg: ASbGncs2bMk9yJ2zaw7FPQ9QTYUSYoTHsRoqfov5OdzVimM9NXdO3NoADonrIA2HOyl cla35Jw3tR1LG9KCkmYT9xfE2pBilN050sL69yGNyvl5WNDWLTF44kaIdqZbyQxgJMeV+MTavCu OMXsXtRgJTpaRIAuL5dHBdy5Z6CTYI6heo27GmwksHNkd2aL7fUmPRQtFpUAQ= X-Google-Smtp-Source: AGHT+IFe9V7N1AM6I97qW2PIFaJOc//5ykQN3qvLTGodBUY1IufUQVVhjk57UdLRVj+E6MV6tHM6KVD9DIXLiSNcilo= X-Received: by 2002:a17:90b:190a:b0:2ff:6e72:b8e9 with SMTP id 98e67ed59e1d1-306a61ee18cmr9849168a91.25.1743943021708; Sun, 06 Apr 2025 05:37:01 -0700 (PDT) MIME-Version: 1.0 From: Motog Plus Date: Sun, 6 Apr 2025 18:06:50 +0530 X-Gm-Features: ATxdqUE7OURBMLtt2ermNcUflW2ToUIEp68YmiR4JDekAEJnMyQK3ET4Els0_hM Message-ID: Subject: =?UTF-8?Q?Memory_Not_Released_After_Batch_Completion_=E2=80=93_Check?= =?UTF-8?Q?pointer=2FBackground_Writer_Behavior_=2C_postgres_15?= To: pgsql-performance@postgresql.org Content-Type: multipart/alternative; boundary="000000000000514d0b06321b60c9" List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk --000000000000514d0b06321b60c9 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi All, I=E2=80=99m running PostgreSQL on an EC2 c5.4xlarge Ubuntu instance with th= e following specs: 32 GB RAM 1.2 TB disk 16 vCPUs Pgpool-II Configuration: max_pool =3D 2 num_init_children =3D 1000 client_idle_limit =3D 300 seconds connection_life_time =3D 300 seconds load_balance_mode =3D on PostgreSQL Configuration: max_connections =3D 3000 checkpoint_timeout =3D 15min checkpoint_completion_target =3D 0.9 shared_buffers =3D 10GB wal_buffers =3D 64MB min_wal_size =3D 80MB max_wal_size =3D 10GB effective_cache_size =3D 20GB work_mem =3D 4MB maintenance_work_mem =3D 1GB bgwriter_delay =3D 200ms bgwriter_lru_multiplier =3D 2 bgwriter_lru_maxpages =3D 100 max_standby_streaming_delay =3D 5s (on standby) I have a primary-standby streaming replication setup, and application modules (running in Kubernetes pods) connect to the database through Pgpool-II using HikariCP. --- Issue Observed: Even when no queries are running and application batch jobs have completed, memory usage remains high(40%)=E2=80=94 particularly attributable to the checkpointer and background writer processes. According to Grafana, memory continues to remain cached. When I manually kill the checkpointer process on the node, memory usage drops immediately(9%). If I don't kill it and start another batch, memory usage increases further =E2=80=94 reaching up to 90%, though under normal conditions it stays around 65=E2=80=9370%. --- System Memory Snapshot (when idle): free -h : total 30 Gi used 12 Gi free 2.2 Gi shared 10Gi buff/cache 27Gi available 18Gi PostgreSQL memory usage from ps: ps -eo pid,rss,cmd | grep postgres | awk '{sum+=3D$2} END {print sum/1024 " MB"}' Output: 25682.4 MB Note: Even though the system is idle for over 50 hours, connections connect, become idle and drop. --- Question: Is it normal for the checkpointer and background writer to hold onto memory like this after workloads finish? Is this related to how PostgreSQL manages shared buffers or dirty pages in a streaming replication setup? Are there any parameters in postgresql.conf or Pgpool that can help ensure memory is better reclaimed when the system is idle? Any guidance would be much appreciated. Best regards, Ramzy --000000000000514d0b06321b60c9 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable

Hi All,

I=E2=80=99m running Postgre= SQL on an EC2 c5.4xlarge Ubuntu instance with the following specs:

32 GB RAM

1.2 TB disk

<= div dir=3D"auto">16 vCPUs


Pgpool-II Configuration:

max_pool =3D 2

num_init_children =3D 1000
client_idle_limit =3D 300 seconds

connection_life_time =3D 300 seconds

load_balance_mode =3D on


PostgreSQL Configuration:

max_connections =3D 3000

checkpoint_timeout =3D 15min

checkpoint_completion_target =3D 0.9

<= /div>
shared_buffers =3D 10GB

<= /div>
wal_buffers =3D 64MB

min_wal_size =3D 80MB

=
max_wal_size =3D 10GB

effective_cache_size =3D 20GB

work_mem =3D 4MB

maintenance_work_mem =3D 1GB

bgwriter_delay =3D 200ms

bgwriter_lru_multiplier =3D 2
<= br>
bgwriter_lru_maxpages =3D 100

max_standby_streaming_delay =3D 5s (on sta= ndby)


I have a primary-standby streaming replication setup, and applica= tion modules (running in Kubernetes pods) connect to the database through P= gpool-II using HikariCP.


---

Issue Observed:

Even when no queries are running and application batch jobs have complete= d, memory usage remains high(40%)=E2=80=94 particularly attributable to the= checkpointer and background writer processes. According to Grafana, memory= continues to remain cached.

When I manually kill the checkpointer process on the node, memory usag= e drops immediately(9%). If I don't kill it and start another batch, me= mory usage increases further =E2=80=94 reaching up to 90%, though under nor= mal conditions it stays around 65=E2=80=9370%.

<= /div>

---

System Memory Snapshot (when idle):

free -h :
total 30 Gi
used 12 Gi
free 2.2 Gi
shared 10Gi
buff/cache 27Gi
available 18Gi
=
PostgreSQL memory usage from ps:
<= br>
ps -eo pid,rss,cmd | grep postgres | awk '{s= um+=3D$2} END {print sum/1024 " MB"}'
=
Output:
25682.4 MB
=

Note: Even though the system = is idle for over 50 hours, connections connect, become idle and drop.
=

---
Question:

Is it normal for the checkpointer and background writer to hold= onto memory like this after workloads finish? Is this related to how Postg= reSQL manages shared buffers or dirty pages in a streaming replication setu= p?

Are there any paramet= ers in postgresql.conf or Pgpool that can help ensure memory is better recl= aimed when the system is idle?

Any guidance would be much appreciated.

<= /div>
Best regards,
Ramzy
--000000000000514d0b06321b60c9--