Message-ID: <0ed6a6f6-e868-41d7-b28e-0a09362c35f1@vondra.me>
Date: Wed, 2 Oct 2024 10:22:43 +0200
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
Subject: Re: Linux OOM killer
To: Laurenz Albe <laurenz.albe@cybertec.at>, Ariel Tejera
 <artejera@gmail.com>, pgsql-bugs@lists.postgresql.org
References: 
 <CAEO-Fc6dPK+oLeaqYbYf8Pjtfrvh+nwtmyOGyWNv83ZQRocfiA@mail.gmail.com>
 <bfffb132207733e9fbf5fdabe5e36f4f37f39932.camel@cybertec.at>
Content-Language: en-US
From: Tomas Vondra <tomas@vondra.me>
In-Reply-To: <bfffb132207733e9fbf5fdabe5e36f4f37f39932.camel@cybertec.at>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Archived-At: 
 <https://www.postgresql.org/message-id/0ed6a6f6-e868-41d7-b28e-0a09362c35f1%40vondra.me>
Precedence: bulk


On 10/2/24 06:16, Laurenz Albe wrote:
> On Tue, 2024-10-01 at 12:17 -0600, Ariel Tejera wrote:
>> Hi.  I hope this message finds you well.
>>
>> The issue is that one of our Postgres servers hit a bug and was killed by linux OOM, as
>> shown in the lines below, showing two events:
>>
>> We were able to fix this problem adjusting the server configuration with:
>> enable_memoize = off
>>
>> Our Postgres version is 14.5 
>> Linux AWS linux2 (with diverse concurrent workloads)
>> Ram 32GB
>> Database size 200 GB 
>>
>> This is the first reproducible bug I've found in 20 years using postgres, heavily (!)
>>
>> As this bug is associated with large databases, it is impractical to offer a reproducible example for it.
>> We hope, however, that this report will be of some use for the Postgres project.
> 
> First of all, update to 14.latest.  I find at least one bug fixed in this area:
> https://postgr.es/c/e4b95b9b02, discussed in https://postgr.es/m/83281eed63c74e4f940317186372abfd%40cft.ru
> 
> Then, disable memory overcommit, so that you don't get killed by the OOM killer.
> Then you will get an "out of memory" error and a memory context dump in the log.
> We'd need to see that to figure out if it really is a bug.
> 

FWIW I don't think anyone can investigate this without more information.
In particular, we'd need the query plan triggering the issue, with info
about the schema (which data types, ...) and data sizes. And the memory
context information - either logged during OOM, or collected using gdb.

But yeah, definitely update to newest 14.x first. Chances are this is
already fixed.


regards

-- 
Tomas Vondra