Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1ouuyx-0003Na-S9 for pgsql-general@arkaria.postgresql.org; Tue, 15 Nov 2022 12:24:03 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1ouuyw-00058F-9R for pgsql-general@arkaria.postgresql.org; Tue, 15 Nov 2022 12:24:02 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1ouuyv-000585-V0 for pgsql-general@lists.postgresql.org; Tue, 15 Nov 2022 12:24:01 +0000 Received: from mail.pernau.at ([176.9.98.135]) by magus.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1ouuyt-00016S-H8 for pgsql-general@postgresql.org; Tue, 15 Nov 2022 12:24:01 +0000 Received: from pernau.at (localhost.localdomain [127.0.0.1]) by mail.pernau.at (Postfix) with ESMTPSA id 24FFF2F41854; Tue, 15 Nov 2022 13:23:57 +0100 (CET) MIME-Version: 1.0 Date: Tue, 15 Nov 2022 13:23:56 +0100 From: klaus.mailinglists@pernau.at To: Christoph Moench-Tegeder Cc: pgsql-general@postgresql.org Subject: Re: PANIC: could not flush dirty data: Cannot allocate memory In-Reply-To: References: Message-ID: <7e02611b56aebc62a0ec25bb39687491@pernau.at> X-Sender: klaus.mailinglists@pernau.at Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Transfer-Encoding: 7bit List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk Thanks all for digging into this problem. AFAIU the problem is not related to the memory settings in postgresql.conf. It is the kernel that for whatever reasons report ENOMEM. Correct? Am 2022-11-14 22:54, schrieb Christoph Moench-Tegeder: > ## klaus.mailinglists@pernau.at (klaus.mailinglists@pernau.at): > >> On several servers we see the error message: PANIC: could not flush >> dirty data: Cannot allocate memory > > As far as I can see, that "could not flush dirty data" happens total > three times in the code - there are other places where postgresql could > PANIC on fsync()-and-stuff-related issues, but they have different > messages. > Of these three places, there's an sync_file_range(), an posix_fadvise() > and an msync(), all in src/backend/storage/file/fd.c. "Cannot allocate > memory" would be ENOMEM, which posix_fadvise() does not return (as per > it's docs). So this would be sync_file_range(), which could run out > of memory (as per the manual) or msync() where ENOMEM actually means > "The indicated memory (or part of it) was not mapped". Both cases are > somewhat WTF for this setup. > What filesystem are you running? Filesystem is ext4. VM technology is mixed: VMware, KVM and XEN PV. Kernel is 5.15.0-52-generic. We have not seen this with Ubutnu 18.04 and 20.04 (although we might not have noticed it). I guess upgrading to postgresql 13/14/15 does not help as the problem happens in the kernel. Do you have any advice how to go further? Shall I lookout for certain kernel changes? In the kernel itself or in ext4 changelog? Thanks Klaus