Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1r2Y9B-00HMpD-BX for pgsql-general@arkaria.postgresql.org; Mon, 13 Nov 2023 14:42:41 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.94.2) (envelope-from ) id 1r2Y98-00Fuqo-GY for pgsql-general@arkaria.postgresql.org; Mon, 13 Nov 2023 14:42:38 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1r2Y98-00Fuqg-5e for pgsql-general@lists.postgresql.org; Mon, 13 Nov 2023 14:42:38 +0000 Received: from charmander.telsasoft.com ([50.244.222.1] helo=pryzbyj2023.telsasoft) by magus.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1r2Y95-006ECH-44 for pgsql-general@lists.postgresql.org; Mon, 13 Nov 2023 14:42:36 +0000 Received: by pryzbyj2023.telsasoft (Postfix, from userid 1000) id C713A9296; Mon, 13 Nov 2023 08:42:31 -0600 (CST) Date: Mon, 13 Nov 2023 08:42:31 -0600 From: Justin Pryzby To: pgsql-general@lists.postgresql.org Cc: Christoph Berg , Devrim =?iso-8859-1?B?R/xuZPx6?= Subject: Re: Properly handle OOM death? Message-ID: References: <20230313174301.23n6crb5dpttozup@hjp.at> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <20230313174301.23n6crb5dpttozup@hjp.at> List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk On Mon, Mar 13, 2023 at 06:43:01PM +0100, Peter J. Holzer wrote: > On 2023-03-13 09:21:18 -0800, Israel Brewster wrote: > > I’m running a postgresql 13 database on an Ubuntu 20.04 VM that is a bit more > > memory constrained than I would like, such that every week or so the various > > processes running on the machine will align badly and the OOM killer will kick > > in, killing off postgresql, as per the following journalctl output: > > > > Mar 12 04:04:23 novarupta systemd[1]: postgresql@13-main.service: A process of this unit has been killed by the OOM killer. > > Mar 12 04:04:32 novarupta systemd[1]: postgresql@13-main.service: Failed with result 'oom-kill'. > > Mar 12 04:04:32 novarupta systemd[1]: postgresql@13-main.service: Consumed 5d 17h 48min 24.509s CPU time. > > > > And the service is no longer running. > > I might be misreading this, but it looks to me that systemd detects that > *some* process in the group was killed by the oom killer and stops the > service. Yeah. I found this old message on google. I'm surprised there aren't more, similar complaints about this. It's as Peter said: it (sometimes) causes systemd to actively *stop* the cluster after OOM, when it would've come back online on its own if the init (supervisor) process didn't interfere. My solution was to set: /usr/lib/systemd/system/postgresql@.service OOMPolicy=continue I suggest that the default unit files should do likewise. -- Justin