public inbox for [email protected]help / color / mirror / Atom feed
Error:could not extend file " with FileFallocate(): No space left on device 3+ messages / 3 participants [nested] [flat]
* Error:could not extend file " with FileFallocate(): No space left on device @ 2024-09-10 13:11 =?iso-8859-2?Q?Pecs=F6k_J=E1n?= <[email protected]> 2024-09-11 09:55 ` Re: Error:could not extend file " with FileFallocate(): No space left on device Alvaro Herrera <[email protected]> 0 siblings, 1 reply; 3+ messages in thread From: =?iso-8859-2?Q?Pecs=F6k_J=E1n?= @ 2024-09-10 13:11 UTC (permalink / raw) To: [email protected] <[email protected]> Dear community, After upgrade of Posgres from version 13.5 to 16.2 we experience following error: could not extend file "pg_tblspc/16401/PG_16_202307071/17820/3968302971" with FileFallocate(): No space left on device We cannot easily replicate problem. It happens at randomly every 1-2 weeks of intensive query computation. Was there some changes in space allocation from Posgres 13.5 to Posgres 16.2? Database has size 91TB and has 27TB more space available. ^ permalink raw reply [nested|flat] 3+ messages in thread
* Re: Error:could not extend file " with FileFallocate(): No space left on device 2024-09-10 13:11 Error:could not extend file " with FileFallocate(): No space left on device =?iso-8859-2?Q?Pecs=F6k_J=E1n?= <[email protected]> @ 2024-09-11 09:55 ` Alvaro Herrera <[email protected]> 2024-09-11 10:44 ` Re: Error:could not extend file " with FileFallocate(): No space left on device Thomas Munro <[email protected]> 0 siblings, 1 reply; 3+ messages in thread From: Alvaro Herrera @ 2024-09-11 09:55 UTC (permalink / raw) To: Pecsök Ján <[email protected]>; +Cc: [email protected] <[email protected]>; Thomas Munro <[email protected]>; Andres Freund <[email protected]> Hello Ján, On 2024-Sep-10, Pecsök Ján wrote: > After upgrade of Posgres from version 13.5 to 16.2 we experience following error: > could not extend file "pg_tblspc/16401/PG_16_202307071/17820/3968302971" with FileFallocate(): No space left on device > > We cannot easily replicate problem. It happens at randomly every 1-2 weeks of intensive query computation. > Was there some changes in space allocation from Posgres 13.5 to Posgres 16.2? > > Database has size 91TB and has 27TB more space available. Yes, there were some changes in that area. I have a report from somebody running EPAS 16 which has a problem that looks pretty much the same as yours -- and the code is essentially identical. I gave them the attached patch, hoping that it would shed some light ... but so far, we've been unable to capture any useful intel. I'm going to propose this patch for the next set of minors, but that's in November, so if you're in a hurry and want to risk rebuilding Postgres and see if you get any better error messages with it than with the original, here it is. (Note that the patch doesn't change any behavior, it just report more things when a problem occurs.) I'm CCing Thomas Munro and Andres Freund, who authored the new code. -- Álvaro Herrera 48°01'N 7°57'E — https://www.EnterpriseDB.com/ "El sudor es la mejor cura para un pensamiento enfermo" (Bardia) Attachments: [text/x-diff] 0001-Add-some-debugging-around-mdzeroextend.patch (2.7K, 2-0001-Add-some-debugging-around-mdzeroextend.patch) download | inline diff: From bf5b7b137687a90cb3523f815c015eb5d95fd85c Mon Sep 17 00:00:00 2001 From: Alvaro Herrera <[email protected]> Date: Fri, 5 Jul 2024 12:51:42 +0200 Subject: [PATCH] Add some debugging around mdzeroextend This is all new code, so it could have bugs. --- src/backend/storage/file/fd.c | 7 +++++++ src/backend/storage/smgr/md.c | 12 ++++++++---- 2 files changed, 15 insertions(+), 4 deletions(-) diff --git a/src/backend/storage/file/fd.c b/src/backend/storage/file/fd.c index 16b3e8f9058..a5db22b9b37 100644 --- a/src/backend/storage/file/fd.c +++ b/src/backend/storage/file/fd.c @@ -2347,6 +2347,13 @@ retry: */ if (returnCode != EINVAL && returnCode != EOPNOTSUPP) return -1; + + if (returnCode != EOPNOTSUPP) + ereport(WARNING, + errcode(ERRCODE_WARNING), + errmsg("could not allocate additional %lld bytes from position %lld in file \"%s\", retrying by zeroing: %m", + (long long) amount, (long long) offset, + VfdCache[file].fileName)); #endif return FileZero(file, offset, amount, wait_event_info); diff --git a/src/backend/storage/smgr/md.c b/src/backend/storage/smgr/md.c index fdecbad1709..ac5e325f8ec 100644 --- a/src/backend/storage/smgr/md.c +++ b/src/backend/storage/smgr/md.c @@ -580,15 +580,17 @@ mdzeroextend(SMgrRelation reln, ForkNumber forknum, if (numblocks > 8) { int ret; + off_t addbytes = (off_t) BLCKSZ * numblocks; ret = FileFallocate(v->mdfd_vfd, - seekpos, (off_t) BLCKSZ * numblocks, + seekpos, addbytes, WAIT_EVENT_DATA_FILE_EXTEND); if (ret != 0) { ereport(ERROR, errcode_for_file_access(), - errmsg("could not extend file \"%s\" with FileFallocate(): %m", + errmsg("could not allocate additional %lld bytes from position %lld in file \"%s\": %m", + (long long) addbytes, (long long) seekpos, FilePathName(v->mdfd_vfd)), errhint("Check free disk space.")); } @@ -596,6 +598,7 @@ mdzeroextend(SMgrRelation reln, ForkNumber forknum, else { int ret; + off_t addbytes = (off_t) BLCKSZ * numblocks; /* * Even if we don't want to use fallocate, we can still extend a @@ -605,12 +608,13 @@ mdzeroextend(SMgrRelation reln, ForkNumber forknum, * whole length of the extension. */ ret = FileZero(v->mdfd_vfd, - seekpos, (off_t) BLCKSZ * numblocks, + seekpos, addbytes, WAIT_EVENT_DATA_FILE_EXTEND); if (ret < 0) ereport(ERROR, errcode_for_file_access(), - errmsg("could not extend file \"%s\": %m", + errmsg("could not zero additional %lld bytes from position %lld file \"%s\": %m", + (long long) addbytes, (long long) seekpos, FilePathName(v->mdfd_vfd)), errhint("Check free disk space.")); } -- 2.39.2 ^ permalink raw reply [nested|flat] 3+ messages in thread
* Re: Error:could not extend file " with FileFallocate(): No space left on device 2024-09-10 13:11 Error:could not extend file " with FileFallocate(): No space left on device =?iso-8859-2?Q?Pecs=F6k_J=E1n?= <[email protected]> 2024-09-11 09:55 ` Re: Error:could not extend file " with FileFallocate(): No space left on device Alvaro Herrera <[email protected]> @ 2024-09-11 10:44 ` Thomas Munro <[email protected]> 0 siblings, 0 replies; 3+ messages in thread From: Thomas Munro @ 2024-09-11 10:44 UTC (permalink / raw) To: Alvaro Herrera <[email protected]>; +Cc: Pecsök Ján <[email protected]>; [email protected] <[email protected]>; Andres Freund <[email protected]> On Wed, Sep 11, 2024 at 9:56 PM Alvaro Herrera <[email protected]> wrote: > On 2024-Sep-10, Pecsök Ján wrote: > > After upgrade of Posgres from version 13.5 to 16.2 we experience following error: > > could not extend file "pg_tblspc/16401/PG_16_202307071/17820/3968302971" with FileFallocate(): No space left on device > > > > We cannot easily replicate problem. It happens at randomly every 1-2 weeks of intensive query computation. > > Was there some changes in space allocation from Posgres 13.5 to Posgres 16.2? > > > > Database has size 91TB and has 27TB more space available. > > Yes, there were some changes in that area. I have a report from > somebody running EPAS 16 which has a problem that looks pretty much the > same as yours -- and the code is essentially identical. I gave them the > attached patch, hoping that it would shed some light ... but so far, > we've been unable to capture any useful intel. What kernel version and file system are these running on (both cases)? ^ permalink raw reply [nested|flat] 3+ messages in thread
end of thread, other threads:[~2024-09-11 10:44 UTC | newest] Thread overview: 3+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2024-09-10 13:11 Error:could not extend file " with FileFallocate(): No space left on device =?iso-8859-2?Q?Pecs=F6k_J=E1n?= <[email protected]> 2024-09-11 09:55 ` Alvaro Herrera <[email protected]> 2024-09-11 10:44 ` Thomas Munro <[email protected]>
This inbox is served by agora; see mirroring instructions for how to clone and mirror all data and code used for this inbox