Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1svqn4-000HFj-5R for pgsql-bugs@arkaria.postgresql.org; Wed, 02 Oct 2024 04:16:42 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.94.2) (envelope-from ) id 1svqn3-009hpE-1E for pgsql-bugs@arkaria.postgresql.org; Wed, 02 Oct 2024 04:16:41 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1svqn2-009hp6-OL for pgsql-bugs@lists.postgresql.org; Wed, 02 Oct 2024 04:16:40 +0000 Received: from mail-wm1-x331.google.com ([2a00:1450:4864:20::331]) by makus.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.94.2) (envelope-from ) id 1svqmv-00246q-Vg for pgsql-bugs@lists.postgresql.org; Wed, 02 Oct 2024 04:16:39 +0000 Received: by mail-wm1-x331.google.com with SMTP id 5b1f17b1804b1-42cae6bb895so57139255e9.1 for ; Tue, 01 Oct 2024 21:16:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cybertec.at; s=cybertec.at; t=1727842592; x=1728447392; darn=lists.postgresql.org; h=mime-version:user-agent:content-transfer-encoding:references :in-reply-to:date:to:from:subject:message-id:from:to:cc:subject:date :message-id:reply-to; bh=p48jYYtVSNRLvfnjwexQKiu6A2af77SO9FQQ/aGjj84=; b=m/cd51BYsTh2ffDSMfDDpa5IrRQ4meJDiQ4PafYXLvZyEf8FYzYiBQKrVymGg1VtKQ 0FWAXymbjhYkmVSwZN6kywmxVOlRIi/tVHEE0Qd9pSVhpLZDfX8UGBH+wu4kocM2tJwc cCFHP/aQ316q7yBbmhSbkWqQX1BsNtYqP7EEc= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727842592; x=1728447392; h=mime-version:user-agent:content-transfer-encoding:references :in-reply-to:date:to:from:subject:message-id:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=p48jYYtVSNRLvfnjwexQKiu6A2af77SO9FQQ/aGjj84=; b=XiO3lN4rOrt4kiGcih6KJdzt/gedgyvWCRm6Bk2UyneKgPzD2Uo2SsLWkVWal6GBAT mrwrBxwwQy3zLHk7RTH24d/D348kUzfzGrdfFV4NlIONkAr1ZUkBR36A9y89Tp2x/q3n z5JVcabkRwSEZXjWtHAGYO1BEYT9knPAsJ/3uBH4Zbpc9A5ZJe3zyz3R/tvWQRL4EsvM ENSETzLVNOEdClYqw4UCSndKEUOUdsmFTfoxtGdAxlpxHccpMudfOoYAycVh8JUt9aaY Kfs0HikIe7SlRaKHDdBbhOkyjHj6W4DfNeUgOWktbcQj7qBtzfjme+JkzpZxhCKp0I8q k0xQ== X-Forwarded-Encrypted: i=1; AJvYcCUM4RQnNLjbrD3GhittpD5vPMIUChXpx8QZfmDcH8DaAD/BoBuBTBESewEHbp6PTgp7PVFFRMPAUY51@lists.postgresql.org X-Gm-Message-State: AOJu0YwyeEe1hBJgM7Z2mjFwaWQEdkQo9Z/XeUiveug/rzI+l72txgDL ilSCXsq4E7rI7Ilqq1q+aO1rk4PVuSFliOvJMoowC1PG7TyeCxH6GmIif+AVED4= X-Google-Smtp-Source: AGHT+IEn3UWhlPX/yH+2RcssbnG1GSxXycs0Ife+pBCHLvirXXvsecWyi9oWDezDiJr31T2HzOVAiw== X-Received: by 2002:a05:600c:1d08:b0:42c:b9c8:2bb0 with SMTP id 5b1f17b1804b1-42f777b1f6emr12982605e9.4.1727842592172; Tue, 01 Oct 2024 21:16:32 -0700 (PDT) Received: from dynamic-pd01.res.v6.highway.a1.net ([2001:871:260:d007:9e75:bc6:9729:1147]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-42f771a1209sm15306065e9.0.2024.10.01.21.16.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 01 Oct 2024 21:16:31 -0700 (PDT) Message-ID: Subject: Re: Linux OOM killer From: Laurenz Albe To: Ariel Tejera , pgsql-bugs@lists.postgresql.org Date: Wed, 02 Oct 2024 06:16:31 +0200 In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable User-Agent: Evolution 3.52.4 (3.52.4-1.fc40) MIME-Version: 1.0 List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk On Tue, 2024-10-01 at 12:17 -0600, Ariel Tejera wrote: > Hi.=C2=A0 I hope this message finds you well. >=20 > The issue is that one of our Postgres servers hit a bug and was killed by= linux OOM, as > shown in the lines below, showing two events: >=20 > We were able to fix this problem adjusting the server configuration with: > enable_memoize =3D off >=20 > Our Postgres version is 14.5=C2=A0 > Linux AWS linux2 (with diverse concurrent workloads) > Ram 32GB > Database size 200 GB=C2=A0 >=20 > This is the first reproducible bug I've found in 20 years using postgres,= heavily (!) >=20 > As this bug is associated with large databases, it is impractical to offe= r a reproducible example for it. > We hope, however, that this report will be of some use for the Postgres p= roject. First of all, update to 14.latest. I find at least one bug fixed in this a= rea: https://postgr.es/c/e4b95b9b02, discussed in https://postgr.es/m/83281eed63= c74e4f940317186372abfd%40cft.ru Then, disable memory overcommit, so that you don't get killed by the OOM ki= ller. Then you will get an "out of memory" error and a memory context dump in the= log. We'd need to see that to figure out if it really is a bug. It need not be a bug if you run out of memory. It might as well be that yo= u configured PostgreSQL too generously. Yours, Laurenz Albe