From: klaus.mailinglists@pernau.at
To: Christoph Moench-Tegeder <cmt@burggraben.net>
Cc: pgsql-general@postgresql.org
Subject: Re: PANIC: could not flush dirty data: Cannot allocate memory
Date: Tue, 15 Nov 2022 13:23:56 +0100
Message-ID: <7e02611b56aebc62a0ec25bb39687491@pernau.at> (raw)
In-Reply-To: <Y3K4+z5fP9mNElMj@elch.exwg.net>
References: <ec01f7af535c0c38711a317cba2aa310@pernau.at>
<Y3K4+z5fP9mNElMj@elch.exwg.net>
Thanks all for digging into this problem.
AFAIU the problem is not related to the memory settings in
postgresql.conf. It is the kernel that
for whatever reasons report ENOMEM. Correct?
Am 2022-11-14 22:54, schrieb Christoph Moench-Tegeder:
> ## klaus.mailinglists@pernau.at (klaus.mailinglists@pernau.at):
>
>> On several servers we see the error message: PANIC: could not flush
>> dirty data: Cannot allocate memory
>
> As far as I can see, that "could not flush dirty data" happens total
> three times in the code - there are other places where postgresql could
> PANIC on fsync()-and-stuff-related issues, but they have different
> messages.
> Of these three places, there's an sync_file_range(), an posix_fadvise()
> and an msync(), all in src/backend/storage/file/fd.c. "Cannot allocate
> memory" would be ENOMEM, which posix_fadvise() does not return (as per
> it's docs). So this would be sync_file_range(), which could run out
> of memory (as per the manual) or msync() where ENOMEM actually means
> "The indicated memory (or part of it) was not mapped". Both cases are
> somewhat WTF for this setup.
> What filesystem are you running?
Filesystem is ext4. VM technology is mixed: VMware, KVM and XEN PV.
Kernel is 5.15.0-52-generic.
We have not seen this with Ubutnu 18.04 and 20.04 (although we might not
have noticed it).
I guess upgrading to postgresql 13/14/15 does not help as the problem
happens in the kernel.
Do you have any advice how to go further? Shall I lookout for certain
kernel changes? In the kernel itself or in ext4 changelog?
Thanks
Klaus
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients from your mail client; the link fills in
the headers below: reply via email
Some clients, webmail especially, drop In-Reply-To and References from
a link, and the reply then starts a new thread: use the mbox below.
To: pgsql-general@postgresql.org
Cc: klaus.mailinglists@pernau.at, cmt@burggraben.net
Subject: Re: PANIC: could not flush dirty data: Cannot allocate memory
In-Reply-To: <7e02611b56aebc62a0ec25bb39687491@pernau.at>
References: <ec01f7af535c0c38711a317cba2aa310@pernau.at> <Y3K4+z5fP9mNElMj@elch.exwg.net> <7e02611b56aebc62a0ec25bb39687491@pernau.at>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox