Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1lMEgf-000394-J0 for pgsql-bugs@arkaria.postgresql.org; Tue, 16 Mar 2021 18:45:01 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1lMEge-000677-Gg for pgsql-bugs@arkaria.postgresql.org; Tue, 16 Mar 2021 18:45:00 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1lMEge-000670-Ad for pgsql-bugs@lists.postgresql.org; Tue, 16 Mar 2021 18:45:00 +0000 Received: from tamriel.snowman.net ([2001:470:e38f::11]) by makus.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1lMEgc-0004BB-1y for pgsql-bugs@lists.postgresql.org; Tue, 16 Mar 2021 18:44:59 +0000 Received: by tamriel.snowman.net (Postfix, from userid 1000) id 1AEA95F799; Tue, 16 Mar 2021 14:44:56 -0400 (EDT) Date: Tue, 16 Mar 2021 14:44:55 -0400 From: Stephen Frost To: Jelte Fennema Cc: Michael Paquier , duspensky@ya.ru, pgsql-bugs@lists.postgresql.org Subject: Re: BUG #16160: Minor memory leak in case of starting postgres server with SSL encryption Message-ID: <20210316184455.GB20766@tamriel.snowman.net> References: <16160-18367e56e9a28264@postgresql.org> <20191213063915.GE1942@paquier.xyz> <20191214092456.GB70616@paquier.xyz> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="5k3NaXtxvQDwwmiV" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.24 (2015-08-30) List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk --5k3NaXtxvQDwwmiV Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Greetings, * Jelte Fennema (postgres@jeltef.nl) wrote: > We ran into this memory leak on PG11 in production. The lea was determined > to be the root cause of OOM errors we were seeing. There was a combination > of a things that caused this leak to become serious enough for these OOM > errors to happen: OOMs errors should be gracefully handled and PG should continue to function. Was that not the case..? > To clarify the context a bit more if you're not familiar with the details > of vm.overcommit_memory: There's "used" memory and "commited_as" memory. > The copy-on-write memory in all backends is counted towards "commited_as" > memory. "used" memory does not increase for every backend, because it's > copy-on-write and none of the backends write to this memory (since it's > leaked so there's no live pointer to it). Right- and is also why it's certainly important to be monitoring the committed_as value vs the commit limit. > Linux puts a hard limit on commited_as, because we use > vm.overcommit_memory=2 (which means memory overcommitting is disabled). If > we had memory overcommiting enabled, then this memory leak wouldn't be a > real problem. The amount of "used" memory is pretty much negligable. It > only becomes a problem, because it's commited_as is multiplied for every > process and we care about commited_as because of disabled overcommiting. Allowing overcommit, on the other hand, ends up with the Linux OOM Killer running and sending essentially a kill -9 to PG, causing the entire PG instance to crash and have to go through recovery. > It would be great if this could be backpatched to all currently supported > PG versions. The patch is very small, so it should be very little effort I > think. I'd be happy to help with that if that's useful or needed. +1 on back-patching these fixes. -1 on what came across, to me at least, as an argument for allowing overcommit. I realize you didn't explicitly say that, but figured it'd be good for the archives to discuss a bit more about why having overcommit_memory set to 2 is strongly recommended. Without that, runaway queries could lead to the OOM Killer running and the entire PG instance crashing. Thanks, Stephen --5k3NaXtxvQDwwmiV Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQIcBAEBCgAGBQJgUPynAAoJEO1sijiDR2RVbSMQAJ1How+HnH6ERZL4Dt16E1wY z08+ZpkldW1/3q4BJz9Yj30dHrB539PcQH0OSeo6Tu+PTKTnbXnu7rl776LGnez1 hYb1iZ5XFtqZVby5UE4Crezr4gzHDW226La5uU5lz/BoM2jv5nHPkJtRrjfxjOmp HJV3CD7gP9ACM7ngmJkr5IFCZ/DNUMINN/8vspSPvE2glNLrJra3QAmrtoyoqjYR 2oHHxWJfRTyGi0N5IptS55gtCRtN2QUTSG+EaAk7TWvOAaqQDSSUaD1lO95n9jxR tq1xXSJYxzZgP/TkoeaPQLNvBfJDtlOETMXc1OTL0gwDPWSSCAUI4R/gYAiHhBIr R+VFQct/bKJvAMkL1PPhGqOqqKLLRS2fAcZSTxxGaCt5cJrfcXDc0I5W6CY7wfkc SSiw+YY6+cwrYxMtVP4ndT/A0Y5De/j/oegb8DeL9N5MfjYgW3l7ndmWXfnhAfdq MVOmK3XJskZrSKjvzTaKl/B3L5MyGrl00UOtH9/xCotzugnsTzcW/qlFGsdMcmST xqhfLvZ7da6dM0/QBztNRXz0taH9H7CNOGQj5ygdGr0L/KuEmIoab3AiK1lnt3hC e1so5sTogY4KqUrQVwjxLMcOVbLimaAWiwHqmrWDRSzbJiai1SV8eIJF5+zer8CD /ZnAgBkj8mmpdvuUr8QN =ikX5 -----END PGP SIGNATURE----- --5k3NaXtxvQDwwmiV--