public inbox for [email protected]
help / color / mirror / Atom feedKubernetes, cgroups v2 and OOM killer - how to avoid?
2+ messages / 2 participants
[nested] [flat]
* Kubernetes, cgroups v2 and OOM killer - how to avoid?
@ 2025-04-05 11:53 Ancoron Luciferis <[email protected]>
2025-04-07 08:10 ` Re: Kubernetes, cgroups v2 and OOM killer - how to avoid? Laurenz Albe <[email protected]>
0 siblings, 1 reply; 2+ messages in thread
From: Ancoron Luciferis @ 2025-04-05 11:53 UTC (permalink / raw)
To: [email protected]
Hi,
I've been investigating this topic every now and then but to this day
have not come to a setup that consistently leads to a PostgreSQL backend
process receiving an allocation error instead of being killed externally
by the OOM killer.
Why this is a problem for me? Because while applications are accessing
their DBs (multiple services having their own DBs, some high-frequency),
the whole server goes into recovery and kills all backends/connections.
While my applications are written to tolerate that, it also means that
at that time, esp. for the high-frequency apps, events are piling up,
which then leads to a burst as soon as connectivity is restored. This in
turn leads to peaks in resource usage in other places (event store,
in-memory buffers from apps, ...), which sometimes leads to a series of
OOM killer events being triggered, just because some analytics query
went overboard.
Ideally, I'd find a configuration that only terminates one backend but
leaves the others working.
I am wondering whether there is any way to receive a real ENOMEM inside
a cgroup as soon as I try to allocate beyond its memory.max, instead of
relying on the OOM killer.
I know the recommendation is to have vm.overcommit_memory set to 2, but
then that affects all workloads on the host, including critical infra
like the kubelet, CNI, CSI, monitoring, ...
I have already gone through and tested the obvious:
https://www.postgresql.org/docs/current/kernel-resources.html#LINUX-MEMORY-OVERCOMMIT
And yes, I know that Linux cgroups v2 memory.max is not an actual hard
limit:
https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v2.html#memory-interface-files
Any help is greatly appreciated!
Cheers,
Ancoron
^ permalink raw reply [nested|flat] 2+ messages in thread
* Re: Kubernetes, cgroups v2 and OOM killer - how to avoid?
2025-04-05 11:53 Kubernetes, cgroups v2 and OOM killer - how to avoid? Ancoron Luciferis <[email protected]>
@ 2025-04-07 08:10 ` Laurenz Albe <[email protected]>
0 siblings, 0 replies; 2+ messages in thread
From: Laurenz Albe @ 2025-04-07 08:10 UTC (permalink / raw)
To: Ancoron Luciferis <[email protected]>; [email protected]
On Sat, 2025-04-05 at 13:53 +0200, Ancoron Luciferis wrote:
> I've been investigating this topic every now and then but to this day
> have not come to a setup that consistently leads to a PostgreSQL backend
> process receiving an allocation error instead of being killed externally
> by the OOM killer.
>
> Why this is a problem for me? Because while applications are accessing
> their DBs (multiple services having their own DBs, some high-frequency),
> the whole server goes into recovery and kills all backends/connections.
You don't have to explain why that is a problem. It clearly is!
> Ideally, I'd find a configuration that only terminates one backend but
> leaves the others working.
There isn't, but what you really want is:
> I am wondering whether there is any way to receive a real ENOMEM inside
> a cgroup as soon as I try to allocate beyond its memory.max, instead of
> relying on the OOM killer.
>
> I know the recommendation is to have vm.overcommit_memory set to 2, but
> then that affects all workloads on the host, including critical infra
> like the kubelet, CNI, CSI, monitoring, ...
I cannot answer your question, but I'd like to make two suggestions:
1. set the PostgreSQL configuration parameters "work_mem", "shared_buffers",
"maintenance_work_mem" and "max_connections" low enough that you don't
go out of memory. A crash is bad, but a query failing with an "out of
memory" error isn't nice either.
2. If you want to run PostgreSQL seriously in Kubernetes, put all PostgreSQL
pods on a dedicated host machine where you can disable memory overcommit.
Yours,
Laurenz Albe
^ permalink raw reply [nested|flat] 2+ messages in thread
end of thread, other threads:[~2025-04-07 08:10 UTC | newest]
Thread overview: 2+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2025-04-05 11:53 Kubernetes, cgroups v2 and OOM killer - how to avoid? Ancoron Luciferis <[email protected]>
2025-04-07 08:10 ` Laurenz Albe <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox