From: Adrian Klaver <adrian.klaver@aklaver.com>
To: Israel Brewster <ijbrewster@alaska.edu>
To: PostgreSQL Mailing Lists <pgsql-general@postgresql.org>
Subject: Re: Properly handle OOM death?
Date: Mon, 13 Mar 2023 10:28:12 -0700
Message-ID: <17d66a4c-4b05-106e-b2c7-e2babbd31de7@aklaver.com> (raw)
In-Reply-To: <E1525E23-A751-41C6-AD65-B3BF6C982B06@alaska.edu>
References: <E1525E23-A751-41C6-AD65-B3BF6C982B06@alaska.edu>
On 3/13/23 10:21 AM, Israel Brewster wrote:
> I’m running a postgresql 13 database on an Ubuntu 20.04 VM that is a bit
> more memory constrained than I would like, such that every week or so
> the various processes running on the machine will align badly and the
> OOM killer will kick in, killing off postgresql, as per the following
> journalctl output:
>
> Mar 12 04:04:23 novarupta systemd[1]: postgresql@13-main.service: A
> process of this unit has been killed by the OOM killer.
> Mar 12 04:04:32 novarupta systemd[1]: postgresql@13-main.service: Failed
> with result 'oom-kill'.
> Mar 12 04:04:32 novarupta systemd[1]: postgresql@13-main.service:
> Consumed 5d 17h 48min 24.509s CPU time.
>
> And the service is no longer running.
>
> When this happens, I go in and restart the postgresql service, and
> everything is happy again for the next week or two.
>
> Obviously this is not a good situation. Which leads to two questions:
>
> 1) is there some tweaking I can do in the postgresql config itself to
> prevent the situation from occurring in the first place?
> 2) My first thought was to simply have systemd restart postgresql
> whenever it is killed like this, which is easy enough. Then I looked at
> the default unit file, and found these lines:
>
> # prevent OOM killer from choosing the postmaster (individual backends will
> # reset the score to 0)
> OOMScoreAdjust=-900
> # restarting automatically will prevent "pg_ctlcluster ... stop" from
> working,
> # so we disable it here. Also, the postmaster will restart by itself on most
> # problems anyway, so it is questionable if one wants to enable external
> # automatic restarts.
> #Restart=on-failure
>
> Which seems to imply that the OOM killer should only be killing off
> individual backends, not the entire cluster to begin with - which should
> be fine. And also that adding the restart=on-failure option is probably
> not the greatest idea. Which makes me wonder what is really going on?
You might want to read:
https://www.postgresql.org/docs/current/kernel-resources.html#LINUX-MEMORY-OVERCOMMIT>
> Thanks.
>
> ---
> Israel Brewster
> Software Engineer
> Alaska Volcano Observatory
> Geophysical Institute - UAF
> 2156 Koyukuk Drive
> Fairbanks AK 99775-7320
> Work: 907-474-5172
> cell: 907-328-9145
>
--
Adrian Klaver
adrian.klaver@aklaver.com
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients from your mail client; the link fills in
the headers below: reply via email
Some clients, webmail especially, drop In-Reply-To and References from
a link, and the reply then starts a new thread: use the mbox below.
To: pgsql-general@postgresql.org
Cc: adrian.klaver@aklaver.com, ijbrewster@alaska.edu
Subject: Re: Properly handle OOM death?
In-Reply-To: <17d66a4c-4b05-106e-b2c7-e2babbd31de7@aklaver.com>
References: <E1525E23-A751-41C6-AD65-B3BF6C982B06@alaska.edu> <17d66a4c-4b05-106e-b2c7-e2babbd31de7@aklaver.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox