From: Tom Lane <tgl@sss.pgh.pa.us>
To: Laurenz Albe <laurenz.albe@cybertec.at>
Cc: frank picabia <fpicabia@gmail.com>
Cc: pgsql-admin@postgresql.org
Subject: Re: Error: "could not fork new process for connection: Cannot allocate memory"
Date: Mon, 21 Dec 2020 10:27:18 -0500
Message-ID: <492638.1608564438@sss.pgh.pa.us> (raw)
In-Reply-To: <fce391ac5b00b86921e680ebd37eaa802e51dd39.camel@cybertec.at>
References: <CA+AKB6F6EGnLPFDqm1kcPtyeRWyY92vbfQk-3uFVbrERYu+PJg@mail.gmail.com>
<fce391ac5b00b86921e680ebd37eaa802e51dd39.camel@cybertec.at>
Laurenz Albe <laurenz.albe@cybertec.at> writes:
> On Mon, 2020-12-21 at 09:17 -0400, frank picabia wrote:
>> First of all, we did not run out of memory within the OS. This system is monitored by cacti and
>> Real Used memory never exceeded 30 GB on a server having 64 GB ram plus some swap.
> You probably have to increase the operating system limit for open files
> for the "postgres" user (ulimit -n).
Isn't ulimit -n a per-process, not per-user, limit? Perhaps ulimit -u
(max user processes) would be worth checking, though.
However, I don't think I believe the assertion that the system wasn't
under overall memory pressure. What we see in the quoted log fragment
is two separate postmaster fork-failure reports interspersed with a
memory context map, which has to have been coming out of some other
process because postmaster.c does not dump its contexts when reporting
a fork failure. But there's no reason for a PG process to dump a
context map unless it suffered an ENOMEM allocation failure.
(It'd be interesting to look for the "out of memory" error that presumably
follows the context map, to see if it offers any more info.)
So what we have is fork() being unhappy concurrently with ENOMEM
problems in at least one other process. That smells like overall
memory pressure to me, cacti or no cacti.
If you've got cgroups enabled, or if the whole thing is running
inside a VM, there might be kernel-enforced memory limits somewhere.
regards, tom lane
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-admin@postgresql.org
Cc: tgl@sss.pgh.pa.us, laurenz.albe@cybertec.at, fpicabia@gmail.com
Subject: Re: Error: "could not fork new process for connection: Cannot allocate memory"
In-Reply-To: <492638.1608564438@sss.pgh.pa.us>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox