Feedback-ID: ia2694551:Fastmail
Date: Wed, 11 Sep 2024 11:55:59 +0200
From: Alvaro Herrera <alvherre@alvh.no-ip.org>
To: =?utf-8?B?UGVjc8O2ayBKw6Fu?= <jan.pecsok@profinit.eu>
Cc: 
	"pgsql-general@lists.postgresql.org" <pgsql-general@lists.postgresql.org>, Thomas Munro <thomas.munro@gmail.com>, 
	Andres Freund <andres@anarazel.de>
Subject: Re: Error:could not extend file " with FileFallocate(): No space
 left on device
Message-ID: <202409110955.6njbwzm4ocus@alvherre.pgsql>
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="rjbc3qehavvp44am"
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <AS1PR05MB91059AC8B525910A5FCD6E699F9A2@AS1PR05MB9105.eurprd05.prod.outlook.com>
Archived-At: <https://www.postgresql.org/message-id/202409110955.6njbwzm4ocus%40alvherre.pgsql>
Precedence: bulk


--rjbc3qehavvp44am
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit

Hello Ján,

On 2024-Sep-10, Pecsök Ján wrote:

> After upgrade of Posgres from version 13.5 to 16.2 we experience following error:
> could not extend file "pg_tblspc/16401/PG_16_202307071/17820/3968302971" with FileFallocate(): No space left on device
> 
> We cannot easily replicate problem. It happens at randomly every 1-2 weeks of intensive query computation.
> Was there some changes in space allocation from Posgres 13.5  to Posgres  16.2?
> 
> Database has  size 91TB and has 27TB more space available.

Yes, there were some changes in that area.  I have a report from
somebody running EPAS 16 which has a problem that looks pretty much the
same as yours -- and the code is essentially identical.  I gave them the
attached patch, hoping that it would shed some light ... but so far,
we've been unable to capture any useful intel.

I'm going to propose this patch for the next set of minors, but that's
in November, so if you're in a hurry and want to risk rebuilding
Postgres and see if you get any better error messages with it than with
the original, here it is.  (Note that the patch doesn't change any
behavior, it just report more things when a problem occurs.)

I'm CCing Thomas Munro and Andres Freund, who authored the new code.

-- 
Álvaro Herrera               48°01'N 7°57'E  —  https://www.EnterpriseDB.com/
"El sudor es la mejor cura para un pensamiento enfermo" (Bardia)

--rjbc3qehavvp44am
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
	filename="0001-Add-some-debugging-around-mdzeroextend.patch"

From bf5b7b137687a90cb3523f815c015eb5d95fd85c Mon Sep 17 00:00:00 2001
From: Alvaro Herrera <alvaro.herrera@enterprisedb.com>
Date: Fri, 5 Jul 2024 12:51:42 +0200
Subject: [PATCH] Add some debugging around mdzeroextend

This is all new code, so it could have bugs.
---
 src/backend/storage/file/fd.c |  7 +++++++
 src/backend/storage/smgr/md.c | 12 ++++++++----
 2 files changed, 15 insertions(+), 4 deletions(-)

diff --git a/src/backend/storage/file/fd.c b/src/backend/storage/file/fd.c
index 16b3e8f9058..a5db22b9b37 100644
--- a/src/backend/storage/file/fd.c
+++ b/src/backend/storage/file/fd.c
@@ -2347,6 +2347,13 @@ retry:
 	 */
 	if (returnCode != EINVAL && returnCode != EOPNOTSUPP)
 		return -1;
+
+	if (returnCode != EOPNOTSUPP)
+		ereport(WARNING,
+				errcode(ERRCODE_WARNING),
+				errmsg("could not allocate additional %lld bytes from position %lld in file \"%s\", retrying by zeroing: %m",
+					   (long long) amount, (long long) offset,
+					   VfdCache[file].fileName));
 #endif
 
 	return FileZero(file, offset, amount, wait_event_info);
diff --git a/src/backend/storage/smgr/md.c b/src/backend/storage/smgr/md.c
index fdecbad1709..ac5e325f8ec 100644
--- a/src/backend/storage/smgr/md.c
+++ b/src/backend/storage/smgr/md.c
@@ -580,15 +580,17 @@ mdzeroextend(SMgrRelation reln, ForkNumber forknum,
 		if (numblocks > 8)
 		{
 			int			ret;
+			off_t		addbytes = (off_t) BLCKSZ * numblocks;
 
 			ret = FileFallocate(v->mdfd_vfd,
-								seekpos, (off_t) BLCKSZ * numblocks,
+								seekpos, addbytes,
 								WAIT_EVENT_DATA_FILE_EXTEND);
 			if (ret != 0)
 			{
 				ereport(ERROR,
 						errcode_for_file_access(),
-						errmsg("could not extend file \"%s\" with FileFallocate(): %m",
+						errmsg("could not allocate additional %lld bytes from position %lld in file \"%s\": %m",
+							   (long long) addbytes, (long long) seekpos,
 							   FilePathName(v->mdfd_vfd)),
 						errhint("Check free disk space."));
 			}
@@ -596,6 +598,7 @@ mdzeroextend(SMgrRelation reln, ForkNumber forknum,
 		else
 		{
 			int			ret;
+			off_t		addbytes = (off_t) BLCKSZ * numblocks;
 
 			/*
 			 * Even if we don't want to use fallocate, we can still extend a
@@ -605,12 +608,13 @@ mdzeroextend(SMgrRelation reln, ForkNumber forknum,
 			 * whole length of the extension.
 			 */
 			ret = FileZero(v->mdfd_vfd,
-						   seekpos, (off_t) BLCKSZ * numblocks,
+						   seekpos, addbytes,
 						   WAIT_EVENT_DATA_FILE_EXTEND);
 			if (ret < 0)
 				ereport(ERROR,
 						errcode_for_file_access(),
-						errmsg("could not extend file \"%s\": %m",
+						errmsg("could not zero additional %lld bytes from position %lld file \"%s\": %m",
+							   (long long) addbytes, (long long) seekpos,
 							   FilePathName(v->mdfd_vfd)),
 						errhint("Check free disk space."));
 		}
-- 
2.39.2


--rjbc3qehavvp44am--