Message-ID: <03a450a1ce93e65f160350628b3d2b125750e5b4.camel@cybertec.at>
Subject: Re: Kubernetes, cgroups v2 and OOM killer - how to avoid?
From: Laurenz Albe <laurenz.albe@cybertec.at>
To: Ancoron Luciferis <ancoron.luciferis@googlemail.com>, 
	pgsql-general@lists.postgresql.org
Date: Mon, 07 Apr 2025 10:10:07 +0200
In-Reply-To: <b5a262b6-f33f-4ace-b4c3-0c28ad0369b7@googlemail.com>
References: <b5a262b6-f33f-4ace-b4c3-0c28ad0369b7@googlemail.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
User-Agent: Evolution 3.54.3 (3.54.3-1.fc41) 
MIME-Version: 1.0
Archived-At: <https://www.postgresql.org/message-id/03a450a1ce93e65f160350628b3d2b125750e5b4.camel%40cybertec.at>
Precedence: bulk

On Sat, 2025-04-05 at 13:53 +0200, Ancoron Luciferis wrote:
> I've been investigating this topic every now and then but to this day=20
> have not come to a setup that consistently leads to a PostgreSQL backend=
=20
> process receiving an allocation error instead of being killed externally=
=20
> by the OOM killer.
>=20
> Why this is a problem for me? Because while applications are accessing=
=20
> their DBs (multiple services having their own DBs, some high-frequency),=
=20
> the whole server goes into recovery and kills all backends/connections.

You don't have to explain why that is a problem.  It clearly is!

> Ideally, I'd find a configuration that only terminates one backend but=
=20
> leaves the others working.

There isn't, but what you really want is:

> I am wondering whether there is any way to receive a real ENOMEM inside=
=20
> a cgroup as soon as I try to allocate beyond its memory.max, instead of=
=20
> relying on the OOM killer.
>=20
> I know the recommendation is to have vm.overcommit_memory set to 2, but=
=20
> then that affects all workloads on the host, including critical infra=20
> like the kubelet, CNI, CSI, monitoring, ...

I cannot answer your question, but I'd like to make two suggestions:

1. set the PostgreSQL configuration parameters "work_mem", "shared_buffers"=
,
   "maintenance_work_mem" and "max_connections" low enough that you don't
   go out of memory.  A crash is bad, but a query failing with an "out of
   memory" error isn't nice either.

2. If you want to run PostgreSQL seriously in Kubernetes, put all PostgreSQ=
L
   pods on a dedicated host machine where you can disable memory overcommit=
.

Yours,
Laurenz Albe