public inbox for [email protected]  

help / color / mirror / Atom feed

From: Laurenz Albe <[email protected]>
To: Ancoron Luciferis <[email protected]>
To: [email protected]
Subject: Re: Kubernetes, cgroups v2 and OOM killer - how to avoid?
Date: Mon, 07 Apr 2025 10:10:07 +0200
Message-ID: <[email protected]> (raw)
In-Reply-To: <[email protected]>
References: <[email protected]>

On Sat, 2025-04-05 at 13:53 +0200, Ancoron Luciferis wrote:
> I've been investigating this topic every now and then but to this day 
> have not come to a setup that consistently leads to a PostgreSQL backend 
> process receiving an allocation error instead of being killed externally 
> by the OOM killer.
> 
> Why this is a problem for me? Because while applications are accessing 
> their DBs (multiple services having their own DBs, some high-frequency), 
> the whole server goes into recovery and kills all backends/connections.

You don't have to explain why that is a problem.  It clearly is!

> Ideally, I'd find a configuration that only terminates one backend but 
> leaves the others working.

There isn't, but what you really want is:

> I am wondering whether there is any way to receive a real ENOMEM inside 
> a cgroup as soon as I try to allocate beyond its memory.max, instead of 
> relying on the OOM killer.
> 
> I know the recommendation is to have vm.overcommit_memory set to 2, but 
> then that affects all workloads on the host, including critical infra 
> like the kubelet, CNI, CSI, monitoring, ...

I cannot answer your question, but I'd like to make two suggestions:

1. set the PostgreSQL configuration parameters "work_mem", "shared_buffers",
   "maintenance_work_mem" and "max_connections" low enough that you don't
   go out of memory.  A crash is bad, but a query failing with an "out of
   memory" error isn't nice either.

2. If you want to run PostgreSQL seriously in Kubernetes, put all PostgreSQL
   pods on a dedicated host machine where you can disable memory overcommit.

Yours,
Laurenz Albe

view thread (2+ messages)

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients using the --to and --cc options:
  reply via email

  To: [email protected]
  Cc: [email protected], [email protected], [email protected]
  Subject: Re: Kubernetes, cgroups v2 and OOM killer - how to avoid?
  In-Reply-To: <[email protected]>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox