Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1mXltp-0001xK-CD for pgsql-admin@arkaria.postgresql.org; Tue, 05 Oct 2021 14:58:33 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1mXlto-00074Y-An for pgsql-admin@arkaria.postgresql.org; Tue, 05 Oct 2021 14:58:32 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1mXlto-00074Q-28 for pgsql-admin@lists.postgresql.org; Tue, 05 Oct 2021 14:58:32 +0000 Received: from sss.pgh.pa.us ([66.207.139.130]) by magus.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1mXltl-0003Oe-Ev for pgsql-admin@lists.postgresql.org; Tue, 05 Oct 2021 14:58:31 +0000 Received: from sss1.sss.pgh.pa.us (localhost [127.0.0.1]) by sss.pgh.pa.us (8.15.2/8.15.2) with ESMTP id 195EwRc94071313; Tue, 5 Oct 2021 10:58:27 -0400 From: Tom Lane To: =?UTF-8?B?TGF0aWYgZ8O8ZMO8aw==?= cc: pgsql-admin@lists.postgresql.org Subject: Re: Postgresql db crash and recovery mode In-reply-to: References: Comments: In-reply-to =?UTF-8?B?TGF0aWYgZ8O8ZMO8aw==?= message dated "Tue, 05 Oct 2021 17:52:03 +0300" MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-ID: <4071311.1633445907.1@sss.pgh.pa.us> Content-Transfer-Encoding: quoted-printable Date: Tue, 05 Oct 2021 10:58:27 -0400 Message-ID: <4071312.1633445907@sss.pgh.pa.us> List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk =3D?UTF-8?B?TGF0aWYgZ8O8ZMO8aw=3D=3D?=3D writes: > One of our production database crashed and got FATAL: the database syst= em > is in recovery mode. You can find OS version and db version info below. > 2021-10-05 17:01:08.103 +03 20248 LOG: server process (PID 158856) > was terminated by signal 9: Killed That is an external kill. If you didn't do it manually, it's most likely the Linux OOM killer in action. See https://www.postgresql.org/docs/current/kernel-resources.html#LINUX-MEMORY= -OVERCOMMIT regards, tom lane