public inbox for [email protected]
help / color / mirror / Atom feed Memory Not Released After Batch Completion – Checkpointer/Background Writer Behavior , postgres 15
3+ messages / 2 participants
[nested] [flat]
* Memory Not Released After Batch Completion – Checkpointer/Background Writer Behavior , postgres 15
@ 2025-04-06 12:36 Motog Plus <[email protected]>
2025-04-06 14:05 ` Re: Memory Not Released After Batch Completion – Checkpointer/Background Writer Behavior , postgres 15 Евгений Чекан <[email protected]>
0 siblings, 1 reply; 3+ messages in thread
From: Motog Plus @ 2025-04-06 12:36 UTC (permalink / raw)
To: pgsql-performance
Hi All,
I’m running PostgreSQL on an EC2 c5.4xlarge Ubuntu instance with the
following specs:
32 GB RAM
1.2 TB disk
16 vCPUs
Pgpool-II Configuration:
max_pool = 2
num_init_children = 1000
client_idle_limit = 300 seconds
connection_life_time = 300 seconds
load_balance_mode = on
PostgreSQL Configuration:
max_connections = 3000
checkpoint_timeout = 15min
checkpoint_completion_target = 0.9
shared_buffers = 10GB
wal_buffers = 64MB
min_wal_size = 80MB
max_wal_size = 10GB
effective_cache_size = 20GB
work_mem = 4MB
maintenance_work_mem = 1GB
bgwriter_delay = 200ms
bgwriter_lru_multiplier = 2
bgwriter_lru_maxpages = 100
max_standby_streaming_delay = 5s (on standby)
I have a primary-standby streaming replication setup, and application
modules (running in Kubernetes pods) connect to the database through
Pgpool-II using HikariCP.
---
Issue Observed:
Even when no queries are running and application batch jobs have completed,
memory usage remains high(40%)— particularly attributable to the
checkpointer and background writer processes. According to Grafana, memory
continues to remain cached.
When I manually kill the checkpointer process on the node, memory usage
drops immediately(9%). If I don't kill it and start another batch, memory
usage increases further — reaching up to 90%, though under normal
conditions it stays around 65–70%.
---
System Memory Snapshot (when idle):
free -h :
total 30 Gi
used 12 Gi
free 2.2 Gi
shared 10Gi
buff/cache 27Gi
available 18Gi
PostgreSQL memory usage from ps:
ps -eo pid,rss,cmd | grep postgres | awk '{sum+=$2} END {print sum/1024 "
MB"}'
Output:
25682.4 MB
Note: Even though the system is idle for over 50 hours, connections
connect, become idle and drop.
---
Question:
Is it normal for the checkpointer and background writer to hold onto memory
like this after workloads finish? Is this related to how PostgreSQL manages
shared buffers or dirty pages in a streaming replication setup?
Are there any parameters in postgresql.conf or Pgpool that can help ensure
memory is better reclaimed when the system is idle?
Any guidance would be much appreciated.
Best regards,
Ramzy
^ permalink raw reply [nested|flat] 3+ messages in thread
* Re: Memory Not Released After Batch Completion – Checkpointer/Background Writer Behavior , postgres 15
2025-04-06 12:36 Memory Not Released After Batch Completion – Checkpointer/Background Writer Behavior , postgres 15 Motog Plus <[email protected]>
@ 2025-04-06 14:05 ` Евгений Чекан <[email protected]>
2025-04-06 16:02 ` Re: Memory Not Released After Batch Completion – Checkpointer/Background Writer Behavior , postgres 15 Motog Plus <[email protected]>
0 siblings, 1 reply; 3+ messages in thread
From: Евгений Чекан @ 2025-04-06 14:05 UTC (permalink / raw)
To: Motog Plus <[email protected]>; +Cc: pgsql-performance
Checkpointer process holds the shared_buffers memory and it is the expected behaviour. Even during idle, shared buffers are there for reads and future modifications. You have SB configured to be 10GB and it will stay allocated throughout the whole uptime of the server. This is totally OK and should help (generally) with the performance.
The “idle” thing is slightly less straightforward, as checkpointer and writer try to spread the load in time.
As for memory climbing to 90%, you might want to reduce shared buffers to something less than 1/3 of your RAM and see if it helps.
But again, if there’s no issues with system availability and health then you should not be bothered imo
> 6 апр. 2025 г., в 14:37, Motog Plus <[email protected]> написал(а):
>
>
>
> Hi All,
>
> I’m running PostgreSQL on an EC2 c5.4xlarge Ubuntu instance with the following specs:
>
> 32 GB RAM
>
> 1.2 TB disk
>
> 16 vCPUs
>
>
> Pgpool-II Configuration:
>
> max_pool = 2
>
> num_init_children = 1000
>
> client_idle_limit = 300 seconds
>
> connection_life_time = 300 seconds
>
> load_balance_mode = on
>
>
> PostgreSQL Configuration:
>
> max_connections = 3000
>
> checkpoint_timeout = 15min
>
> checkpoint_completion_target = 0.9
>
> shared_buffers = 10GB
>
> wal_buffers = 64MB
>
> min_wal_size = 80MB
>
> max_wal_size = 10GB
>
> effective_cache_size = 20GB
>
> work_mem = 4MB
>
> maintenance_work_mem = 1GB
>
> bgwriter_delay = 200ms
>
> bgwriter_lru_multiplier = 2
>
> bgwriter_lru_maxpages = 100
>
> max_standby_streaming_delay = 5s (on standby)
>
>
> I have a primary-standby streaming replication setup, and application modules (running in Kubernetes pods) connect to the database through Pgpool-II using HikariCP.
>
>
> ---
>
> Issue Observed:
>
> Even when no queries are running and application batch jobs have completed, memory usage remains high(40%)— particularly attributable to the checkpointer and background writer processes. According to Grafana, memory continues to remain cached.
>
> When I manually kill the checkpointer process on the node, memory usage drops immediately(9%). If I don't kill it and start another batch, memory usage increases further — reaching up to 90%, though under normal conditions it stays around 65–70%.
>
>
> ---
>
> System Memory Snapshot (when idle):
>
> free -h :
> total 30 Gi
> used 12 Gi
> free 2.2 Gi
> shared 10Gi
> buff/cache 27Gi
> available 18Gi
> PostgreSQL memory usage from ps:
>
> ps -eo pid,rss,cmd | grep postgres | awk '{sum+=$2} END {print sum/1024 " MB"}'
>
> Output:
> 25682.4 MB
>
> Note: Even though the system is idle for over 50 hours, connections connect, become idle and drop.
>
> ---
>
> Question:
>
> Is it normal for the checkpointer and background writer to hold onto memory like this after workloads finish? Is this related to how PostgreSQL manages shared buffers or dirty pages in a streaming replication setup?
>
> Are there any parameters in postgresql.conf or Pgpool that can help ensure memory is better reclaimed when the system is idle?
>
> Any guidance would be much appreciated.
>
> Best regards,
> Ramzy
^ permalink raw reply [nested|flat] 3+ messages in thread
* Re: Memory Not Released After Batch Completion – Checkpointer/Background Writer Behavior , postgres 15
2025-04-06 12:36 Memory Not Released After Batch Completion – Checkpointer/Background Writer Behavior , postgres 15 Motog Plus <[email protected]>
2025-04-06 14:05 ` Re: Memory Not Released After Batch Completion – Checkpointer/Background Writer Behavior , postgres 15 Евгений Чекан <[email protected]>
@ 2025-04-06 16:02 ` Motog Plus <[email protected]>
0 siblings, 0 replies; 3+ messages in thread
From: Motog Plus @ 2025-04-06 16:02 UTC (permalink / raw)
To: Евгений Чекан <[email protected]>; +Cc: pgsql-performance
Thank you so much for your prompt response.
Regards,
Ramzy
On Sun, Apr 6, 2025, 19:35 Евгений Чекан <[email protected]> wrote:
> Checkpointer process holds the shared_buffers memory and it is the
> expected behaviour. Even during idle, shared buffers are there for reads
> and future modifications. You have SB configured to be 10GB and it will
> stay allocated throughout the whole uptime of the server. This is totally
> OK and should help (generally) with the performance.
> The “idle” thing is slightly less straightforward, as checkpointer and
> writer try to spread the load in time.
> As for memory climbing to 90%, you might want to reduce shared buffers to
> something less than 1/3 of your RAM and see if it helps.
> But again, if there’s no issues with system availability and health then
> you should not be bothered imo
>
> > 6 апр. 2025 г., в 14:37, Motog Plus <[email protected]> написал(а):
> >
> >
> >
> > Hi All,
> >
> > I’m running PostgreSQL on an EC2 c5.4xlarge Ubuntu instance with the
> following specs:
> >
> > 32 GB RAM
> >
> > 1.2 TB disk
> >
> > 16 vCPUs
> >
> >
> > Pgpool-II Configuration:
> >
> > max_pool = 2
> >
> > num_init_children = 1000
> >
> > client_idle_limit = 300 seconds
> >
> > connection_life_time = 300 seconds
> >
> > load_balance_mode = on
> >
> >
> > PostgreSQL Configuration:
> >
> > max_connections = 3000
> >
> > checkpoint_timeout = 15min
> >
> > checkpoint_completion_target = 0.9
> >
> > shared_buffers = 10GB
> >
> > wal_buffers = 64MB
> >
> > min_wal_size = 80MB
> >
> > max_wal_size = 10GB
> >
> > effective_cache_size = 20GB
> >
> > work_mem = 4MB
> >
> > maintenance_work_mem = 1GB
> >
> > bgwriter_delay = 200ms
> >
> > bgwriter_lru_multiplier = 2
> >
> > bgwriter_lru_maxpages = 100
> >
> > max_standby_streaming_delay = 5s (on standby)
> >
> >
> > I have a primary-standby streaming replication setup, and application
> modules (running in Kubernetes pods) connect to the database through
> Pgpool-II using HikariCP.
> >
> >
> > ---
> >
> > Issue Observed:
> >
> > Even when no queries are running and application batch jobs have
> completed, memory usage remains high(40%)— particularly attributable to the
> checkpointer and background writer processes. According to Grafana, memory
> continues to remain cached.
> >
> > When I manually kill the checkpointer process on the node, memory usage
> drops immediately(9%). If I don't kill it and start another batch, memory
> usage increases further — reaching up to 90%, though under normal
> conditions it stays around 65–70%.
> >
> >
> > ---
> >
> > System Memory Snapshot (when idle):
> >
> > free -h :
> > total 30 Gi
> > used 12 Gi
> > free 2.2 Gi
> > shared 10Gi
> > buff/cache 27Gi
> > available 18Gi
> > PostgreSQL memory usage from ps:
> >
> > ps -eo pid,rss,cmd | grep postgres | awk '{sum+=$2} END {print sum/1024
> " MB"}'
> >
> > Output:
> > 25682.4 MB
> >
> > Note: Even though the system is idle for over 50 hours, connections
> connect, become idle and drop.
> >
> > ---
> >
> > Question:
> >
> > Is it normal for the checkpointer and background writer to hold onto
> memory like this after workloads finish? Is this related to how PostgreSQL
> manages shared buffers or dirty pages in a streaming replication setup?
> >
> > Are there any parameters in postgresql.conf or Pgpool that can help
> ensure memory is better reclaimed when the system is idle?
> >
> > Any guidance would be much appreciated.
> >
> > Best regards,
> > Ramzy
>
^ permalink raw reply [nested|flat] 3+ messages in thread
end of thread, other threads:[~2025-04-06 16:02 UTC | newest]
Thread overview: 3+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2025-04-06 12:36 Memory Not Released After Batch Completion – Checkpointer/Background Writer Behavior , postgres 15 Motog Plus <[email protected]>
2025-04-06 14:05 ` Евгений Чекан <[email protected]>
2025-04-06 16:02 ` Motog Plus <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox