pg.ddx.io pgsql-bugs@postgresql.org mailing list archive help / color / mirror / Atom feed From: Tom Lane <tgl@sss.pgh.pa.us>
To: Juan José SantamarÃa Flecha <juanjo.santamaria@gmail.com>
Cc: egashira.yusuke@fujitsu.com, PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
Subject: Re: BUG #17254: Crash with 0xC0000409 in pg_stat_statements when pg_stat_tmp\pgss_query_texts.stat exceeded 2GB.
Date: Sun, 31 Oct 2021 13:07:42 -0400
Message-ID: <975808.1635700062@sss.pgh.pa.us> (raw )
In-Reply-To: <856857.1635627048@sss.pgh.pa.us >
References: <17254-a926c89dc03375c2@postgresql.org >
<CAC+AXB21yF41p4fWgUOkG1PBA-Ezw-8U0Q=Bint9OQ56qyMPYw@mail.gmail.com >
<CAC+AXB0fozNPMgRTP+PgFMrGcSn2zER0zgcrEbRV0NvpZqmE=A@mail.gmail.com >
<CAC+AXB2tWxoZgj6BnoQaxiWw=vKBxUPtyBn7d8jhHRqxv7xRRw@mail.gmail.com >
<840436.1635611203@sss.pgh.pa.us >
<CAC+AXB2vCUOj6x8drBVUVCgjyy4ZFUu-TFBuNVW=7Emh44EC=w@mail.gmail.com >
<856857.1635627048@sss.pgh.pa.us >
I wrote:
> =?UTF-8?Q?Juan_Jos=C3=A9_Santamar=C3=ADa_Flecha?= <juanjo.santamaria@gmail.com> writes:
>> On Sat, Oct 30, 2021 at 6:26 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
>>> I think instead, we need to turn the subsequent one-off read() call into a
>>> loop that reads no more than INT_MAX bytes at a time. It'd be possible
>>> to restrict that to Windows, but probably no harm in doing it the same
>>> way everywhere.
>> Seems reasonable to me, can such a change be back-patched?
> Don't see why not.
Here's a quick patch for that. I don't have any ability to check it
on Windows, but the logic is easy to verify by reducing the arbitrary
constant to something small. (I used 1GB, not INT_MAX, because I figured
we ought to read in multiples of a filesystem block if possible.)
regards, tom lane
Attachments:
avoid-windows-failure-on-huge-read.patch (text/x-diff, 2.3K) download
diff --git a/contrib/pg_stat_statements/pg_stat_statements.c b/contrib/pg_stat_statements/pg_stat_statements.c
index 07fe0e7cda..726ba59e2b 100644
--- a/contrib/pg_stat_statements/pg_stat_statements.c
+++ b/contrib/pg_stat_statements/pg_stat_statements.c
@@ -2125,6 +2125,7 @@ qtext_load_file(Size *buffer_size)
char *buf;
int fd;
struct stat stat;
+ Size nread;
fd = OpenTransientFile(PGSS_TEXT_FILE, O_RDONLY | PG_BINARY);
if (fd < 0)
@@ -2165,23 +2166,35 @@ qtext_load_file(Size *buffer_size)
}
/*
- * OK, slurp in the file. If we get a short read and errno doesn't get
- * set, the reason is probably that garbage collection truncated the file
- * since we did the fstat(), so we don't log a complaint --- but we don't
- * return the data, either, since it's most likely corrupt due to
- * concurrent writes from garbage collection.
+ * OK, slurp in the file. Windows fails if we try to read more than
+ * INT_MAX bytes at once, and other platforms might not like that either,
+ * so read a very large file in 1GB segments.
*/
- errno = 0;
- if (read(fd, buf, stat.st_size) != stat.st_size)
+ nread = 0;
+ while (nread < stat.st_size)
{
- if (errno)
- ereport(LOG,
- (errcode_for_file_access(),
- errmsg("could not read file \"%s\": %m",
- PGSS_TEXT_FILE)));
- free(buf);
- CloseTransientFile(fd);
- return NULL;
+ int toread = Min(1024 * 1024 * 1024, stat.st_size - nread);
+
+ /*
+ * If we get a short read and errno doesn't get set, the reason is
+ * probably that garbage collection truncated the file since we did
+ * the fstat(), so we don't log a complaint --- but we don't return
+ * the data, either, since it's most likely corrupt due to concurrent
+ * writes from garbage collection.
+ */
+ errno = 0;
+ if (read(fd, buf + nread, toread) != toread)
+ {
+ if (errno)
+ ereport(LOG,
+ (errcode_for_file_access(),
+ errmsg("could not read file \"%s\": %m",
+ PGSS_TEXT_FILE)));
+ free(buf);
+ CloseTransientFile(fd);
+ return NULL;
+ }
+ nread += toread;
}
if (CloseTransientFile(fd) != 0)
@@ -2189,7 +2202,7 @@ qtext_load_file(Size *buffer_size)
(errcode_for_file_access(),
errmsg("could not close file \"%s\": %m", PGSS_TEXT_FILE)));
- *buffer_size = stat.st_size;
+ *buffer_size = nread;
return buf;
}
=
view thread (11+ messages) latest in thread
Message-ID: <975808.1635700062@sss.pgh.pa.us>
Permalink: ../975808.1635700062@sss.pgh.pa.us/
Also on: postgresql.org/message-id/975808.1635700062@sss.pgh.pa.us
copy link · copy postgr.es · copy git am
reply Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-bugs@postgresql.org
Cc: tgl@sss.pgh.pa.us, juanjo.santamaria@gmail.com, pgsql-bugs@lists.postgresql.org
Subject: Re: BUG #17254: Crash with 0xC0000409 in pg_stat_statements when pg_stat_tmp\pgss_query_texts.stat exceeded 2GB.
In-Reply-To: <975808.1635700062@sss.pgh.pa.us>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox