public inbox for [email protected]
help / color / mirror / Atom feedBUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work.
22+ messages / 5 participants
[nested] [flat]
* BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work.
@ 2022-03-25 07:52 PG Bug reporting form <[email protected]>
2022-03-26 05:46 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Michael Paquier <[email protected]>
0 siblings, 1 reply; 22+ messages in thread
From: PG Bug reporting form @ 2022-03-25 07:52 UTC (permalink / raw)
To: [email protected]; +Cc: [email protected]
The following bug has been logged on the website:
Bug reference: 17448
Logged by: Okano Naoki
Email address: [email protected]
PostgreSQL version: 14.2
Operating system: Windows
Description:
With huge_pages = on, the postgres process does not appear to use large
pages.
I checked with VMMap if the large pages are used in the following
environment.
Environment
PostgreSQL version: 14.2
Operating system : Windows 10 20H2
On this page (*) says that in Windows 10, version 1703 and later OS
versions,
you must specify the FILE_MAP_LARGE_PAGES flag with the MapViewOfFile
function
to map large pages.
I think it seems to be the cause that MapViewOfFile() in
src/backend/port/win32_shmem.c
does not specify FILE_MAP_LARGE_PAGES flag.
(*) MapViewOfFileEx function
https://docs.microsoft.com/en-us/windows/win32/api/memoryapi/nf-memoryapi-mapviewoffileex
FILE_MAP_LARGE_PAGES
Starting with Windows 10, version 1703, this flag specifies
that the view should be mapped using large page support.
The size of the view must be a multiple of the size of a large page
reported by the GetLargePageMinimum function,
and the file-mapping object must have been created using the
SEC_LARGE_PAGES option.
If you provide a non-null value for lpBaseAddress,
then the value must be a multiple of GetLargePageMinimum.
^ permalink raw reply [nested|flat] 22+ messages in thread
* Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work.
2022-03-25 07:52 BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. PG Bug reporting form <[email protected]>
@ 2022-03-26 05:46 ` Michael Paquier <[email protected]>
2022-03-26 08:24 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Julien Rouhaud <[email protected]>
0 siblings, 1 reply; 22+ messages in thread
From: Michael Paquier @ 2022-03-26 05:46 UTC (permalink / raw)
To: [email protected]; [email protected]
On Fri, Mar 25, 2022 at 07:52:57AM +0000, PG Bug reporting form wrote:
> On this page (*) says that in Windows 10, version 1703 and later OS
> versions,
> you must specify the FILE_MAP_LARGE_PAGES flag with the MapViewOfFile
> function
> to map large pages.
>
> I think it seems to be the cause that MapViewOfFile() in
> src/backend/port/win32_shmem.c
> does not specify FILE_MAP_LARGE_PAGES flag.
Hmm. Okay. A patch would be straight-forward, as we could just
assign the optional flag in a separate variable at the beginning of
PGSharedMemoryCreate(), similarly to flProtect when we find out that
large pages can be used, then pass it down to MapViewOfFileEx(). I
don't have a Windows 10 machine as recent as that at hand, though..
Perhaps the CI uses Windows machines that would allow to test and
check that, with some logs magically added to debug things.
--
Michael
Attachments:
[application/pgp-signature] signature.asc (833B, ../../[email protected]/2-signature.asc)
download
^ permalink raw reply [nested|flat] 22+ messages in thread
* Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work.
2022-03-25 07:52 BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. PG Bug reporting form <[email protected]>
2022-03-26 05:46 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Michael Paquier <[email protected]>
@ 2022-03-26 08:24 ` Julien Rouhaud <[email protected]>
2022-03-26 09:03 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Michael Paquier <[email protected]>
2022-03-26 11:07 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Thomas Munro <[email protected]>
0 siblings, 2 replies; 22+ messages in thread
From: Julien Rouhaud @ 2022-03-26 08:24 UTC (permalink / raw)
To: Michael Paquier <[email protected]>; +Cc: [email protected]; [email protected]
On Sat, Mar 26, 2022 at 02:46:45PM +0900, Michael Paquier wrote:
> On Fri, Mar 25, 2022 at 07:52:57AM +0000, PG Bug reporting form wrote:
> > On this page (*) says that in Windows 10, version 1703 and later OS
> > versions,
> > you must specify the FILE_MAP_LARGE_PAGES flag with the MapViewOfFile
> > function
> > to map large pages.
> >
> > I think it seems to be the cause that MapViewOfFile() in
> > src/backend/port/win32_shmem.c
> > does not specify FILE_MAP_LARGE_PAGES flag.
>
> I don't have a Windows 10 machine as recent as that at hand, though..
I have a Windows 10 apparently version 20H2 (the versioning doesn't make any
sense) with all needed to compile postgres at hand. I can have a look next
week.
^ permalink raw reply [nested|flat] 22+ messages in thread
* Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work.
2022-03-25 07:52 BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. PG Bug reporting form <[email protected]>
2022-03-26 05:46 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Michael Paquier <[email protected]>
2022-03-26 08:24 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Julien Rouhaud <[email protected]>
@ 2022-03-26 09:03 ` Michael Paquier <[email protected]>
1 sibling, 0 replies; 22+ messages in thread
From: Michael Paquier @ 2022-03-26 09:03 UTC (permalink / raw)
To: Julien Rouhaud <[email protected]>; +Cc: [email protected]; [email protected]
On Sat, Mar 26, 2022 at 04:24:08PM +0800, Julien Rouhaud wrote:
> I have a Windows 10 apparently version 20H2 (the versioning doesn't make any
> sense) with all needed to compile postgres at hand. I can have a look next
> week.
Thanks!
--
Michael
Attachments:
[application/pgp-signature] signature.asc (833B, ../../[email protected]/2-signature.asc)
download
^ permalink raw reply [nested|flat] 22+ messages in thread
* Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work.
2022-03-25 07:52 BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. PG Bug reporting form <[email protected]>
2022-03-26 05:46 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Michael Paquier <[email protected]>
2022-03-26 08:24 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Julien Rouhaud <[email protected]>
@ 2022-03-26 11:07 ` Thomas Munro <[email protected]>
2022-03-26 11:33 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Michael Paquier <[email protected]>
2022-03-30 07:54 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Michael Paquier <[email protected]>
2022-09-16 12:51 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Michael Paquier <[email protected]>
1 sibling, 3 replies; 22+ messages in thread
From: Thomas Munro @ 2022-03-26 11:07 UTC (permalink / raw)
To: Julien Rouhaud <[email protected]>; +Cc: Michael Paquier <[email protected]>; [email protected]; PostgreSQL mailing lists <[email protected]>
On Sat, Mar 26, 2022 at 9:24 PM Julien Rouhaud <[email protected]> wrote:
> On Sat, Mar 26, 2022 at 02:46:45PM +0900, Michael Paquier wrote:
> > On Fri, Mar 25, 2022 at 07:52:57AM +0000, PG Bug reporting form wrote:
> > > On this page (*) says that in Windows 10, version 1703 and later OS
> > > versions,
> > > you must specify the FILE_MAP_LARGE_PAGES flag with the MapViewOfFile
> > > function
> > > to map large pages.
> > >
> > > I think it seems to be the cause that MapViewOfFile() in
> > > src/backend/port/win32_shmem.c
> > > does not specify FILE_MAP_LARGE_PAGES flag.
> >
> > I don't have a Windows 10 machine as recent as that at hand, though..
>
> I have a Windows 10 apparently version 20H2 (the versioning doesn't make any
> sense) with all needed to compile postgres at hand. I can have a look next
> week.
There are traces of method to the madness: It's basically YYMM, but
then after 2004 they switched to H1 and H2 (first/second half of the
year) instead of MM, perhaps to avoid confusion with YYYY format year.
Note also that Windows 10 has a 21H2 and Windows 11 has a 21H2.
Hmm, so all versions of Windows that our current coding worked on were
EOL'd 6 months after PostgreSQL 11 came out with huge_pages support
for Windows:
https://en.wikipedia.org/wiki/Windows_10_version_history
Some question I have: is FILE_MAP_LARGE PAGES a macro? We claim to
support all those ancient zombie OSes like Windows 7, or maybe it's
even XP for 11, and this has to be back-patched to 11, so we might
need to make it conditional. But conditional on what? For example,
does something like the attached work (untested)? What happens if a <
1703 kernel sees this flag, does it reject it or ignore it?
Attachments:
[application/x-patch] 0001-Fix-huge_pages-on-current-Windows.patch (1.8K, ../../CA+hUKGLwOSf-QwVtCcKx9xvugN2BSzEBBeD5DNyTXgcSRp5sTw@mail.gmail.com/2-0001-Fix-huge_pages-on-current-Windows.patch)
download | inline diff:
From baf3e7ed938bed9d16bdaae8ab1e798c7da20a52 Mon Sep 17 00:00:00 2001
From: Thomas Munro <[email protected]>
Date: Sat, 26 Mar 2022 22:44:02 +1300
Subject: [PATCH] Fix huge_pages on current Windows.
Since Windows 10 1703, it's necessary to pass a flag to MapViewOfFile()
to enable large pages at map time.
Back-patch to 11 where huge_pages for Windows landed.
Reported-by: Okano Naoki <[email protected]>
Discussion: https://postgr.es/m/17448-0a96583a67edb1f7%40postgresql.org
---
src/backend/port/win32_shmem.c | 8 +++++++-
1 file changed, 7 insertions(+), 1 deletion(-)
diff --git a/src/backend/port/win32_shmem.c b/src/backend/port/win32_shmem.c
index 6cf69411db..1664784891 100644
--- a/src/backend/port/win32_shmem.c
+++ b/src/backend/port/win32_shmem.c
@@ -216,6 +216,7 @@ PGSharedMemoryCreate(Size size,
SIZE_T largePageSize = 0;
Size orig_size = size;
DWORD flProtect = PAGE_READWRITE;
+ DWORD desiredAccess;
ShmemProtectiveRegion = VirtualAlloc(NULL, PROTECTIVE_REGION_SIZE,
MEM_RESERVE, PAGE_NOACCESS);
@@ -353,12 +354,17 @@ retry:
if (!CloseHandle(hmap))
elog(LOG, "could not close handle to shared memory: error code %lu", GetLastError());
+ desiredAccess = FILE_MAP_WRITE | FILE_MAP_READ;
+#ifdef FILE_MAP_LARGE_PAGES
+ if (flProtect & SEC_LARGE_PAGES)
+ desiredAccess |= FILE_MAP_LARGE_PAGES;
+#endif
/*
* Get a pointer to the new shared memory segment. Map the whole segment
* at once, and let the system decide on the initial address.
*/
- memAddress = MapViewOfFileEx(hmap2, FILE_MAP_WRITE | FILE_MAP_READ, 0, 0, 0, NULL);
+ memAddress = MapViewOfFileEx(hmap2, desiredAccess, 0, 0, 0, NULL);
if (!memAddress)
ereport(FATAL,
(errmsg("could not create shared memory segment: error code %lu", GetLastError()),
--
2.30.2
^ permalink raw reply [nested|flat] 22+ messages in thread
* Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work.
2022-03-25 07:52 BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. PG Bug reporting form <[email protected]>
2022-03-26 05:46 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Michael Paquier <[email protected]>
2022-03-26 08:24 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Julien Rouhaud <[email protected]>
2022-03-26 11:07 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Thomas Munro <[email protected]>
@ 2022-03-26 11:33 ` Michael Paquier <[email protected]>
2 siblings, 0 replies; 22+ messages in thread
From: Michael Paquier @ 2022-03-26 11:33 UTC (permalink / raw)
To: Thomas Munro <[email protected]>; +Cc: Julien Rouhaud <[email protected]>; [email protected]; PostgreSQL mailing lists <[email protected]>
On Sun, Mar 27, 2022 at 12:07:57AM +1300, Thomas Munro wrote:
> Some question I have: is FILE_MAP_LARGE PAGES a macro? We claim to
> support all those ancient zombie OSes like Windows 7, or maybe it's
> even XP for 11, and this has to be back-patched to 11, so we might
> need to make it conditional. But conditional on what? For example,
> does something like the attached work (untested)? What happens if a <
> 1703 kernel sees this flag, does it reject it or ignore it?
Good question. I would choose a soft approach here and not insist if
the flag was not known at compilation time, but we could also take a
more aggressive approach and hardcode a value. Anyway, it seems to me
that the correct solution here would be to compile the code with a
PG_FILE_MAP_LARGE_PAGES that checks if the flag exists at compile
time, and we would set it at run time if we know that we are on a
version of Windows that supports it. IsWindowsVersionOrGreater()
should be able to do the job.
--
Michael
Attachments:
[application/pgp-signature] signature.asc (833B, ../../[email protected]/2-signature.asc)
download
^ permalink raw reply [nested|flat] 22+ messages in thread
* Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work.
2022-03-25 07:52 BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. PG Bug reporting form <[email protected]>
2022-03-26 05:46 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Michael Paquier <[email protected]>
2022-03-26 08:24 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Julien Rouhaud <[email protected]>
2022-03-26 11:07 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Thomas Munro <[email protected]>
@ 2022-03-30 07:54 ` Michael Paquier <[email protected]>
2022-03-31 04:59 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Julien Rouhaud <[email protected]>
2 siblings, 1 reply; 22+ messages in thread
From: Michael Paquier @ 2022-03-30 07:54 UTC (permalink / raw)
To: Thomas Munro <[email protected]>; +Cc: Julien Rouhaud <[email protected]>; [email protected]; PostgreSQL mailing lists <[email protected]>
On Sun, Mar 27, 2022 at 12:07:57AM +1300, Thomas Munro wrote:
> There are traces of method to the madness: It's basically YYMM, but
> then after 2004 they switched to H1 and H2 (first/second half of the
> year) instead of MM, perhaps to avoid confusion with YYYY format year.
> Note also that Windows 10 has a 21H2 and Windows 11 has a 21H2.
>
> Some question I have: is FILE_MAP_LARGE PAGES a macro? We claim to
> support all those ancient zombie OSes like Windows 7, or maybe it's
> even XP for 11, and this has to be back-patched to 11, so we might
> need to make it conditional. But conditional on what? For example,
> does something like the attached work (untested)? What happens if a <
> 1703 kernel sees this flag, does it reject it or ignore it?
I don't have an answer about how much Windows gets angry if we pass
down to MapViewOfFileEx() the flag FILE_MAP_LARGE_PAGES when running
the code on a version of Windows that does not support it.
Anyway, I think that we could just play it safe. See for example the
attached, where I use PG_FILE_MAP_LARGE_PAGES at compile time to find
if the value is set. Then, at run-time, I am just relying on
IsWindowsVersionOrGreater() to do the job, something useful when
huge_pages=on as I guess we should fail hard if we did not know about
FILE_MAP_LARGE_PAGES at compile-time, but try to use huge pages at run
time with version >= 10.0.1703.
Perhaps there is a better thing to do?
--
Michael
Attachments:
[text/x-diff] win32-hugepages-michael.patch (2.7K, ../../YkQMyhsP7xgLDDT%[email protected]/2-win32-hugepages-michael.patch)
download | inline diff:
diff --git a/src/backend/port/win32_shmem.c b/src/backend/port/win32_shmem.c
index 6cf69411db..49b9fa0322 100644
--- a/src/backend/port/win32_shmem.c
+++ b/src/backend/port/win32_shmem.c
@@ -12,6 +12,8 @@
*/
#include "postgres.h"
+#include <versionhelpers.h>
+
#include "miscadmin.h"
#include "storage/dsm.h"
#include "storage/ipc.h"
@@ -39,6 +41,17 @@
#define PROTECTIVE_REGION_SIZE (10 * WIN32_STACK_RLIMIT)
void *ShmemProtectiveRegion = NULL;
+/*
+ * FILE_MAP_LARGE_PAGES is needed for the support of huge pages, and
+ * it is only available since Windows 10.0.1703. PG_FILE_MAP_LARGE_PAGES
+ * saves its value at compile-time.
+ */
+#ifdef FILE_MAP_LARGE_PAGES
+#define PG_FILE_MAP_LARGE_PAGES FILE_MAP_LARGE_PAGES
+#else
+#define PG_FILE_MAP_LARGE_PAGES 0
+#endif
+
HANDLE UsedShmemSegID = INVALID_HANDLE_VALUE;
void *UsedShmemSegAddr = NULL;
static Size UsedShmemSegSize = 0;
@@ -216,6 +229,7 @@ PGSharedMemoryCreate(Size size,
SIZE_T largePageSize = 0;
Size orig_size = size;
DWORD flProtect = PAGE_READWRITE;
+ DWORD desiredAccess;
ShmemProtectiveRegion = VirtualAlloc(NULL, PROTECTIVE_REGION_SIZE,
MEM_RESERVE, PAGE_NOACCESS);
@@ -258,6 +272,22 @@ PGSharedMemoryCreate(Size size,
}
}
+ /*
+ * If this binary has been compiled without FILE_MAP_LARGE_PAGES
+ * available and huge pages are wanted, there is nothing do do
+ * if running on a version older than 10.0.1703 as this flag is a
+ * requirement to allow the use of huge pages.
+ */
+ if (huge_pages == HUGE_PAGES_ON &&
+ PG_FILE_MAP_LARGE_PAGES == 0 &&
+ IsWindowsVersionOrGreater(10, 0, 1703))
+ {
+ ereport(FATAL,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("the processor does not support large pages"),
+ errdetail("The code has been compiled without huge page support for this version of Windows.")));
+ }
+
retry:
#ifdef _WIN64
size_high = size >> 32;
@@ -353,12 +383,21 @@ retry:
if (!CloseHandle(hmap))
elog(LOG, "could not close handle to shared memory: error code %lu", GetLastError());
+ desiredAccess = FILE_MAP_WRITE | FILE_MAP_READ;
+
+ /*
+ * Set large pages if wanted. FILE_MAP_LARGE_PAGES is needed when
+ * running at least Windows 10.0.1703.
+ */
+ if ((flProtect & SEC_LARGE_PAGES) != 0 &&
+ IsWindowsVersionOrGreater(10, 0, 1703))
+ desiredAccess |= PG_FILE_MAP_LARGE_PAGES;
/*
* Get a pointer to the new shared memory segment. Map the whole segment
* at once, and let the system decide on the initial address.
*/
- memAddress = MapViewOfFileEx(hmap2, FILE_MAP_WRITE | FILE_MAP_READ, 0, 0, 0, NULL);
+ memAddress = MapViewOfFileEx(hmap2, desiredAccess, 0, 0, 0, NULL);
if (!memAddress)
ereport(FATAL,
(errmsg("could not create shared memory segment: error code %lu", GetLastError()),
[application/pgp-signature] signature.asc (833B, ../../YkQMyhsP7xgLDDT%[email protected]/3-signature.asc)
download
^ permalink raw reply [nested|flat] 22+ messages in thread
* Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work.
2022-03-25 07:52 BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. PG Bug reporting form <[email protected]>
2022-03-26 05:46 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Michael Paquier <[email protected]>
2022-03-26 08:24 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Julien Rouhaud <[email protected]>
2022-03-26 11:07 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Thomas Munro <[email protected]>
2022-03-30 07:54 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Michael Paquier <[email protected]>
@ 2022-03-31 04:59 ` Julien Rouhaud <[email protected]>
2022-03-31 07:42 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Michael Paquier <[email protected]>
0 siblings, 1 reply; 22+ messages in thread
From: Julien Rouhaud @ 2022-03-31 04:59 UTC (permalink / raw)
To: Michael Paquier <[email protected]>; +Cc: Thomas Munro <[email protected]>; [email protected]; PostgreSQL mailing lists <[email protected]>
Hi,
On Wed, Mar 30, 2022 at 04:54:50PM +0900, Michael Paquier wrote:
> On Sun, Mar 27, 2022 at 12:07:57AM +1300, Thomas Munro wrote:
> > There are traces of method to the madness: It's basically YYMM, but
> > then after 2004 they switched to H1 and H2 (first/second half of the
> > year) instead of MM, perhaps to avoid confusion with YYYY format year.
> > Note also that Windows 10 has a 21H2 and Windows 11 has a 21H2.
> >
> > Some question I have: is FILE_MAP_LARGE PAGES a macro? We claim to
> > support all those ancient zombie OSes like Windows 7, or maybe it's
> > even XP for 11, and this has to be back-patched to 11, so we might
> > need to make it conditional. But conditional on what? For example,
> > does something like the attached work (untested)? What happens if a <
> > 1703 kernel sees this flag, does it reject it or ignore it?
>
> I don't have an answer about how much Windows gets angry if we pass
> down to MapViewOfFileEx() the flag FILE_MAP_LARGE_PAGES when running
> the code on a version of Windows that does not support it.
No idea either, and I don't have old enough Windows machine available to try.
> Anyway, I think that we could just play it safe. See for example the
> attached, where I use PG_FILE_MAP_LARGE_PAGES at compile time to find
> if the value is set. Then, at run-time, I am just relying on
> IsWindowsVersionOrGreater() to do the job, something useful when
> huge_pages=on as I guess we should fail hard if we did not know about
> FILE_MAP_LARGE_PAGES at compile-time, but try to use huge pages at run
> time with version >= 10.0.1703.
That approach seems sensible. For reference the versionhelpers.h seems to be
available starting with VS 2013 / v120, which is ok since that the oldest
version we support AFAICS.
After *a lot of time* I could finally test this patch. For the record I could
never find a way to allow 'Lock pages in memory' on the Windows 10 home I have,
so I tried on my Windows 11 evaluation I also had around (version 21H2, so it
should be recent enough). For the record on this one there was gpedit
available, but then I got a 1450 error, and didn't find any information on how
to reserve huge pages or something like that on Windows. So I just configured
shared_buffers to 10MB, which should still be big enough to need multiple huge
pages, and it seems to work:
postgres=# select version();
version
---------------------------------------------------------------
PostgreSQL 15devel, compiled by Visual C++ build 1929, 64-bit
(1 row)
postgres=# show huge_pages;
huge_pages
------------
on
(1 row)
Now, I also have the exact same result without the patch applied so it's hard
to know whether it had any impact at all. Unfortunately, I didn't find any
information on how to check if "large pages" are used and/or by which program.
^ permalink raw reply [nested|flat] 22+ messages in thread
* Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work.
2022-03-25 07:52 BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. PG Bug reporting form <[email protected]>
2022-03-26 05:46 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Michael Paquier <[email protected]>
2022-03-26 08:24 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Julien Rouhaud <[email protected]>
2022-03-26 11:07 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Thomas Munro <[email protected]>
2022-03-30 07:54 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Michael Paquier <[email protected]>
2022-03-31 04:59 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Julien Rouhaud <[email protected]>
@ 2022-03-31 07:42 ` Michael Paquier <[email protected]>
2022-03-31 08:00 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Julien Rouhaud <[email protected]>
0 siblings, 1 reply; 22+ messages in thread
From: Michael Paquier @ 2022-03-31 07:42 UTC (permalink / raw)
To: Julien Rouhaud <[email protected]>; +Cc: Thomas Munro <[email protected]>; [email protected]; PostgreSQL mailing lists <[email protected]>
On Thu, Mar 31, 2022 at 12:59:08PM +0800, Julien Rouhaud wrote:
> That approach seems sensible. For reference the versionhelpers.h seems to be
> available starting with VS 2013 / v120, which is ok since that the oldest
> version we support AFAICS.
Hmm. This points out to a different problem. The oldest version of
MSVC supported on v10 and v11 is VS2005, so a backpatch means that
those checks would need to be tweaked if we'd want to be perfectly
compatible. But I'd rather not enter in this game category, limiting
this patch to v12~ :)
> Now, I also have the exact same result without the patch applied so it's hard
> to know whether it had any impact at all. Unfortunately, I didn't find any
> information on how to check if "large pages" are used and/or by which program.
Okano-san has mentioned VMMap upthread:
https://docs.microsoft.com/en-us/sysinternals/downloads/vmmap
--
Michael
Attachments:
[application/pgp-signature] signature.asc (833B, ../../[email protected]/2-signature.asc)
download
^ permalink raw reply [nested|flat] 22+ messages in thread
* Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work.
2022-03-25 07:52 BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. PG Bug reporting form <[email protected]>
2022-03-26 05:46 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Michael Paquier <[email protected]>
2022-03-26 08:24 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Julien Rouhaud <[email protected]>
2022-03-26 11:07 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Thomas Munro <[email protected]>
2022-03-30 07:54 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Michael Paquier <[email protected]>
2022-03-31 04:59 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Julien Rouhaud <[email protected]>
2022-03-31 07:42 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Michael Paquier <[email protected]>
@ 2022-03-31 08:00 ` Julien Rouhaud <[email protected]>
2022-03-31 10:46 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Julien Rouhaud <[email protected]>
0 siblings, 1 reply; 22+ messages in thread
From: Julien Rouhaud @ 2022-03-31 08:00 UTC (permalink / raw)
To: Michael Paquier <[email protected]>; +Cc: Thomas Munro <[email protected]>; [email protected]; PostgreSQL mailing lists <[email protected]>
On Thu, Mar 31, 2022 at 04:42:37PM +0900, Michael Paquier wrote:
> On Thu, Mar 31, 2022 at 12:59:08PM +0800, Julien Rouhaud wrote:
> > That approach seems sensible. For reference the versionhelpers.h seems to be
> > available starting with VS 2013 / v120, which is ok since that the oldest
> > version we support AFAICS.
>
> Hmm. This points out to a different problem. The oldest version of
> MSVC supported on v10 and v11 is VS2005, so a backpatch means that
> those checks would need to be tweaked if we'd want to be perfectly
> compatible. But I'd rather not enter in this game category, limiting
> this patch to v12~ :)
Ah, I indeed haven't check in back branches. v12 isn't that bad.
> > Now, I also have the exact same result without the patch applied so it's hard
> > to know whether it had any impact at all. Unfortunately, I didn't find any
> > information on how to check if "large pages" are used and/or by which program.
>
> Okano-san has mentioned VMMap upthread:
> https://docs.microsoft.com/en-us/sysinternals/downloads/vmmap
Yes, I totally missed that. Thomas Munro also mentioned it off-list, and
also found some reference [1] indicating that large pages should show up as
"Locked WS". I tested with and without the patch and in both case I don't see
any "Locked WS" usage. I also get the same Page Table size, which seems
consistent with large pages not being used. Now, this is a vm running with
virtualbox and we're not entirely sure that huge pages can be allocated with
it. I wish I could test on my windows 10 machine as it's not virtualized, but
I can't give the required privileges.
[1] https://aloiskraus.wordpress.com/2016/10/03/windows-10-memory-compression-and-more/
^ permalink raw reply [nested|flat] 22+ messages in thread
* Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work.
2022-03-25 07:52 BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. PG Bug reporting form <[email protected]>
2022-03-26 05:46 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Michael Paquier <[email protected]>
2022-03-26 08:24 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Julien Rouhaud <[email protected]>
2022-03-26 11:07 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Thomas Munro <[email protected]>
2022-03-30 07:54 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Michael Paquier <[email protected]>
2022-03-31 04:59 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Julien Rouhaud <[email protected]>
2022-03-31 07:42 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Michael Paquier <[email protected]>
2022-03-31 08:00 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Julien Rouhaud <[email protected]>
@ 2022-03-31 10:46 ` Julien Rouhaud <[email protected]>
2022-03-31 11:03 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Julien Rouhaud <[email protected]>
2022-04-01 05:54 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Michael Paquier <[email protected]>
0 siblings, 2 replies; 22+ messages in thread
From: Julien Rouhaud @ 2022-03-31 10:46 UTC (permalink / raw)
To: Michael Paquier <[email protected]>; +Cc: Thomas Munro <[email protected]>; [email protected]; PostgreSQL mailing lists <[email protected]>
On Thu, Mar 31, 2022 at 04:00:55PM +0800, Julien Rouhaud wrote:
> > Okano-san has mentioned VMMap upthread:
> > https://docs.microsoft.com/en-us/sysinternals/downloads/vmmap
>
> Yes, I totally missed that. Thomas Munro also mentioned it off-list, and
> also found some reference [1] indicating that large pages should show up as
> "Locked WS". I tested with and without the patch and in both case I don't see
> any "Locked WS" usage. I also get the same Page Table size, which seems
> consistent with large pages not being used. Now, this is a vm running with
> virtualbox and we're not entirely sure that huge pages can be allocated with
> it. I wish I could test on my windows 10 machine as it's not virtualized, but
> I can't give the required privileges.
>
> [1] https://aloiskraus.wordpress.com/2016/10/03/windows-10-memory-compression-and-more/
So, after more digging it turns out that the patch is supposed to work. If I
force using the PG_FILE_MAP_LARGE_PAGES, postgres starts and I do see "Locked
WS" usage with VMMap, with a size in the order of magnitude of my
shared_buffers.
What is apparently not working on my VM is IsWindowsVersionOrGreater(10, 0,
1703). I added some debug around to check what GetVersionEx() [2] is saying,
and I get:
dwMajorVersion == 6
dwMinorVersion == 2
dwBuildNumber == 9200
While winver.exe on the same vm says windows 11, version 21H2, build 22000.493.
I'm therefore extremely confused. The documentation of
IsWindowsVersionOrGreater() at [3] is also highly confusing:
> TRUE if the specified version matches, or is greater than, the version of the
> current Windows OS; otherwise, FALSE.
Isn't that supposed to be the opposite?
[2] https://docs.microsoft.com/en-us/windows/win32/api/sysinfoapi/nf-sysinfoapi-getversionexa
https://docs.microsoft.com/en-us/windows/win32/api/winnt/ns-winnt-osversioninfoexa
[3] https://docs.microsoft.com/en-us/windows/win32/api/versionhelpers/nf-versionhelpers-iswindowsversion...
^ permalink raw reply [nested|flat] 22+ messages in thread
* Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work.
2022-03-25 07:52 BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. PG Bug reporting form <[email protected]>
2022-03-26 05:46 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Michael Paquier <[email protected]>
2022-03-26 08:24 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Julien Rouhaud <[email protected]>
2022-03-26 11:07 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Thomas Munro <[email protected]>
2022-03-30 07:54 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Michael Paquier <[email protected]>
2022-03-31 04:59 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Julien Rouhaud <[email protected]>
2022-03-31 07:42 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Michael Paquier <[email protected]>
2022-03-31 08:00 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Julien Rouhaud <[email protected]>
2022-03-31 10:46 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Julien Rouhaud <[email protected]>
@ 2022-03-31 11:03 ` Julien Rouhaud <[email protected]>
2022-04-01 05:59 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Michael Paquier <[email protected]>
2022-04-26 04:54 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Julien Rouhaud <[email protected]>
1 sibling, 2 replies; 22+ messages in thread
From: Julien Rouhaud @ 2022-03-31 11:03 UTC (permalink / raw)
To: Michael Paquier <[email protected]>; +Cc: Thomas Munro <[email protected]>; [email protected]; PostgreSQL mailing lists <[email protected]>
On Thu, Mar 31, 2022 at 06:46:59PM +0800, Julien Rouhaud wrote:
> On Thu, Mar 31, 2022 at 04:00:55PM +0800, Julien Rouhaud wrote:
> > > Okano-san has mentioned VMMap upthread:
> > > https://docs.microsoft.com/en-us/sysinternals/downloads/vmmap
> >
> > Yes, I totally missed that. Thomas Munro also mentioned it off-list, and
> > also found some reference [1] indicating that large pages should show up as
> > "Locked WS". I tested with and without the patch and in both case I don't see
> > any "Locked WS" usage. I also get the same Page Table size, which seems
> > consistent with large pages not being used. Now, this is a vm running with
> > virtualbox and we're not entirely sure that huge pages can be allocated with
> > it. I wish I could test on my windows 10 machine as it's not virtualized, but
> > I can't give the required privileges.
> >
> > [1] https://aloiskraus.wordpress.com/2016/10/03/windows-10-memory-compression-and-more/
>
> So, after more digging it turns out that the patch is supposed to work. If I
> force using the PG_FILE_MAP_LARGE_PAGES, postgres starts and I do see "Locked
> WS" usage with VMMap, with a size in the order of magnitude of my
> shared_buffers.
>
> What is apparently not working on my VM is IsWindowsVersionOrGreater(10, 0,
> 1703). I added some debug around to check what GetVersionEx() [2] is saying,
> and I get:
>
> dwMajorVersion == 6
> dwMinorVersion == 2
> dwBuildNumber == 9200
>
> While winver.exe on the same vm says windows 11, version 21H2, build 22000.493.
So, what GetVersionEx returns is actually "it depends", and this is documented:
> With the release of Windows 8.1, the behavior of the GetVersionEx API has
> changed in the value it will return for the operating system version. The
> value returned by the GetVersionEx function now depends on how the
> application is manifested.
>
> Applications not manifested for Windows 8.1 or Windows 10 will return the
> Windows 8 OS version value (6.2). Once an application is manifested for a
> given operating system version, GetVersionEx will always return the version
> that the application is manifested for in future releases. To manifest your
> applications for Windows 8.1 or Windows 10, refer to Targeting your
> application for Windows.
There's no such indication on IsWindowsVersionOrGreater(), but after seeing
various comments on forums from angry people, it may be a hint that it behaves
similarly. I'm not sure what to do at this point, maybe just always use the
flag (the PG_ version which may be 0), hoping that hopefully windows won't
define it if it can't handle it?
^ permalink raw reply [nested|flat] 22+ messages in thread
* Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work.
2022-03-25 07:52 BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. PG Bug reporting form <[email protected]>
2022-03-26 05:46 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Michael Paquier <[email protected]>
2022-03-26 08:24 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Julien Rouhaud <[email protected]>
2022-03-26 11:07 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Thomas Munro <[email protected]>
2022-03-30 07:54 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Michael Paquier <[email protected]>
2022-03-31 04:59 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Julien Rouhaud <[email protected]>
2022-03-31 07:42 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Michael Paquier <[email protected]>
2022-03-31 08:00 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Julien Rouhaud <[email protected]>
2022-03-31 10:46 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Julien Rouhaud <[email protected]>
2022-03-31 11:03 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Julien Rouhaud <[email protected]>
@ 2022-04-01 05:59 ` Michael Paquier <[email protected]>
2022-04-01 06:19 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Julien Rouhaud <[email protected]>
1 sibling, 1 reply; 22+ messages in thread
From: Michael Paquier @ 2022-04-01 05:59 UTC (permalink / raw)
To: Julien Rouhaud <[email protected]>; +Cc: Thomas Munro <[email protected]>; [email protected]; PostgreSQL mailing lists <[email protected]>
On Thu, Mar 31, 2022 at 07:03:09PM +0800, Julien Rouhaud wrote:
> There's no such indication on IsWindowsVersionOrGreater(), but after seeing
> various comments on forums from angry people, it may be a hint that it behaves
> similarly. I'm not sure what to do at this point, maybe just always use the
> flag (the PG_ version which may be 0), hoping that hopefully windows won't
> define it if it can't handle it?
Looking at the internals of versionhelpers.h, would it work to use as
arguments for IsWindowsVersionOrGreater() the following, in this
order? As of:
- HIBYTE(_WIN32_WINNT_WINTHRESHOLD)
- LOBYTE(_WIN32_WINNT_WINTHRESHOLD)
- 1703
Just to drop an idea.
--
Michael
Attachments:
[application/pgp-signature] signature.asc (833B, ../../[email protected]/2-signature.asc)
download
^ permalink raw reply [nested|flat] 22+ messages in thread
* Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work.
2022-03-25 07:52 BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. PG Bug reporting form <[email protected]>
2022-03-26 05:46 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Michael Paquier <[email protected]>
2022-03-26 08:24 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Julien Rouhaud <[email protected]>
2022-03-26 11:07 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Thomas Munro <[email protected]>
2022-03-30 07:54 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Michael Paquier <[email protected]>
2022-03-31 04:59 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Julien Rouhaud <[email protected]>
2022-03-31 07:42 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Michael Paquier <[email protected]>
2022-03-31 08:00 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Julien Rouhaud <[email protected]>
2022-03-31 10:46 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Julien Rouhaud <[email protected]>
2022-03-31 11:03 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Julien Rouhaud <[email protected]>
2022-04-01 05:59 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Michael Paquier <[email protected]>
@ 2022-04-01 06:19 ` Julien Rouhaud <[email protected]>
0 siblings, 0 replies; 22+ messages in thread
From: Julien Rouhaud @ 2022-04-01 06:19 UTC (permalink / raw)
To: Michael Paquier <[email protected]>; +Cc: Thomas Munro <[email protected]>; [email protected]; PostgreSQL mailing lists <[email protected]>
On Fri, Apr 01, 2022 at 02:59:22PM +0900, Michael Paquier wrote:
> On Thu, Mar 31, 2022 at 07:03:09PM +0800, Julien Rouhaud wrote:
> > There's no such indication on IsWindowsVersionOrGreater(), but after seeing
> > various comments on forums from angry people, it may be a hint that it behaves
> > similarly. I'm not sure what to do at this point, maybe just always use the
> > flag (the PG_ version which may be 0), hoping that hopefully windows won't
> > define it if it can't handle it?
>
> Looking at the internals of versionhelpers.h, would it work to use as
> arguments for IsWindowsVersionOrGreater() the following, in this
> order? As of:
> - HIBYTE(_WIN32_WINNT_WINTHRESHOLD)
> - LOBYTE(_WIN32_WINNT_WINTHRESHOLD)
> - 1703
>
> Just to drop an idea.
I will test that in a bit. I still think that at least some API exists to give
the real answer since winver and similar report correct info, I just don't know
what those are.
Note that if you want to test yourself you could use this script [1] using the
evaluation virtual machine [2] to automatically setup a windows 11 environment
with everything needed to compile postgres with a extra dependencies. The
whole process is a bit long though, so I can also give you access to my vm if
you prefer, probably the latency shouldn't be too bad :)
[1] https://github.com/rjuju/pg_msvc_generator/blob/master/bootstrap.ps1
[2] https://developer.microsoft.com/en-us/windows/downloads/virtual-machines/
^ permalink raw reply [nested|flat] 22+ messages in thread
* Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work.
2022-03-25 07:52 BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. PG Bug reporting form <[email protected]>
2022-03-26 05:46 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Michael Paquier <[email protected]>
2022-03-26 08:24 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Julien Rouhaud <[email protected]>
2022-03-26 11:07 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Thomas Munro <[email protected]>
2022-03-30 07:54 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Michael Paquier <[email protected]>
2022-03-31 04:59 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Julien Rouhaud <[email protected]>
2022-03-31 07:42 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Michael Paquier <[email protected]>
2022-03-31 08:00 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Julien Rouhaud <[email protected]>
2022-03-31 10:46 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Julien Rouhaud <[email protected]>
2022-03-31 11:03 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Julien Rouhaud <[email protected]>
@ 2022-04-26 04:54 ` Julien Rouhaud <[email protected]>
2022-04-27 08:13 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Michael Paquier <[email protected]>
2022-07-07 22:38 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Michael Paquier <[email protected]>
1 sibling, 2 replies; 22+ messages in thread
From: Julien Rouhaud @ 2022-04-26 04:54 UTC (permalink / raw)
To: Wilm Hoyer <[email protected]>; +Cc: [email protected]
Hi,
Please keep the list in copy, especially if that's about Windows specific as
I'm definitely not very knowledgeable about it.
On Fri, Apr 01, 2022 at 09:18:03AM +0000, Wilm Hoyer wrote:
>
> If you don't wanna go the manifest way, maybe the RtlGetVersion function is the one you need:
> https://docs.microsoft.com/en-us/windows-hardware/drivers/ddi/wdm/nf-wdm-rtlgetversion?redirectedfro...
Thanks for the info! I tried to use the function but trying to include either
wdm.h or Ntddk.h errors out. Unfortunately I don't know how to look for a file
in Windows so I don't even know if those files are present.
I searched a bit and apparently some people are using this function directly
opening some dll, which seems wrong.
> Another Idea on windows machines would be to use the commandline to execute
> ver in a separate Process and store the result in a file.
That also seems hackish, I don't think that we want to rely on something like
that.
> >> While winver.exe on the same vm says windows 11, version 21H2, build 22000.493.
>
> > So, what GetVersionEx returns is actually "it depends", and this is documented:
>
> >> With the release of Windows 8.1, the behavior of the GetVersionEx API
> >> has changed in the value it will return for the operating system
> >> version. The value returned by the GetVersionEx function now depends
> >> on how the application is manifested.
> >>
> >> Applications not manifested for Windows 8.1 or Windows 10 will return
> >> the Windows 8 OS version value (6.2). Once an application is
> >> manifested for a given operating system version, GetVersionEx will
> >> always return the version that the application is manifested for in
> >> future releases. To manifest your applications for Windows 8.1 or
> >> Windows 10, refer to Targeting your application for Windows.
>
> The documentation is a bit unclear - with the correct functions you should get the:
> Minimum( actualOS-Version, Maximum(Manifested OS Versions))
> The Idea behind, as I understand it, is to better support virtualization and
> backward compatibility - you manifest only Windows 8.1 -> than you always get
> a System that behaves like Windows 8.1 in every aspect. (Every Aspect not
> true in some corner cases due to security patches)
Well, it clearly does *NOT* behave as a Windows 8.1, even if for some reason
large pages relies on security patches.
Their API is entirely useless, so I'm still on the opinion that we should
unconditionally use the FILE_MAP_LARGE_PAGES flag if it's defined and call it a
day.
^ permalink raw reply [nested|flat] 22+ messages in thread
* Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work.
2022-03-25 07:52 BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. PG Bug reporting form <[email protected]>
2022-03-26 05:46 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Michael Paquier <[email protected]>
2022-03-26 08:24 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Julien Rouhaud <[email protected]>
2022-03-26 11:07 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Thomas Munro <[email protected]>
2022-03-30 07:54 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Michael Paquier <[email protected]>
2022-03-31 04:59 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Julien Rouhaud <[email protected]>
2022-03-31 07:42 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Michael Paquier <[email protected]>
2022-03-31 08:00 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Julien Rouhaud <[email protected]>
2022-03-31 10:46 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Julien Rouhaud <[email protected]>
2022-03-31 11:03 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Julien Rouhaud <[email protected]>
2022-04-26 04:54 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Julien Rouhaud <[email protected]>
@ 2022-04-27 08:13 ` Michael Paquier <[email protected]>
1 sibling, 0 replies; 22+ messages in thread
From: Michael Paquier @ 2022-04-27 08:13 UTC (permalink / raw)
To: Julien Rouhaud <[email protected]>; +Cc: Wilm Hoyer <[email protected]>; [email protected]
On Tue, Apr 26, 2022 at 12:54:35PM +0800, Julien Rouhaud wrote:
> I searched a bit and apparently some people are using this function directly
> opening some dll, which seems wrong.
I was wondering about this whole business, and the manifest approach
is a *horrible* design for an API where the goal is to know if your
run-time environment is greater than a given threshold.
>> Another Idea on windows machines would be to use the commandline to execute
>> ver in a separate Process and store the result in a file.
>
> That also seems hackish, I don't think that we want to rely on something like
> that.
Hmm. That depends on the dependency set, I guess. We do that on
Linux at some extent to for large pages in sysv_shmem.c. Perhaps this
could work for Win10 if this avoids the extra loopholes with the
manifests.
> Their API is entirely useless,
This I agree.
> so I'm still on the opinion that we should
> unconditionally use the FILE_MAP_LARGE_PAGES flag if it's defined and call it a
> day.
Are we sure that this is not going to cause failures in environments
where the flag is not supported?
--
Michael
Attachments:
[application/pgp-signature] signature.asc (833B, ../../[email protected]/2-signature.asc)
download
^ permalink raw reply [nested|flat] 22+ messages in thread
* Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work.
2022-03-25 07:52 BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. PG Bug reporting form <[email protected]>
2022-03-26 05:46 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Michael Paquier <[email protected]>
2022-03-26 08:24 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Julien Rouhaud <[email protected]>
2022-03-26 11:07 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Thomas Munro <[email protected]>
2022-03-30 07:54 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Michael Paquier <[email protected]>
2022-03-31 04:59 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Julien Rouhaud <[email protected]>
2022-03-31 07:42 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Michael Paquier <[email protected]>
2022-03-31 08:00 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Julien Rouhaud <[email protected]>
2022-03-31 10:46 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Julien Rouhaud <[email protected]>
2022-03-31 11:03 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Julien Rouhaud <[email protected]>
2022-04-26 04:54 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Julien Rouhaud <[email protected]>
@ 2022-07-07 22:38 ` Michael Paquier <[email protected]>
1 sibling, 0 replies; 22+ messages in thread
From: Michael Paquier @ 2022-07-07 22:38 UTC (permalink / raw)
To: Julien Rouhaud <[email protected]>; +Cc: Wilm Hoyer <[email protected]>; [email protected]
On Tue, Apr 26, 2022 at 12:54:35PM +0800, Julien Rouhaud wrote:
> Their API is entirely useless, so I'm still on the opinion that we should
> unconditionally use the FILE_MAP_LARGE_PAGES flag if it's defined and call it a
> day.
Now that the minimal runtime version is Windows 10 in v16~ thanks to
495ed0e, we could be much more aggressive and do the attached, which
is roughly what Thomas has proposed upthread at the exception of
assuming that FILE_MAP_LARGE_PAGES always exists, because updates are
forced by MS in this environment. We could make it conditional, of
course, with an extra #ifdef painting.
--
Michael
Attachments:
[text/x-diff] v2-0001-Fix-huge_pages-on-current-Windows.patch (1.7K, ../../[email protected]/2-v2-0001-Fix-huge_pages-on-current-Windows.patch)
download | inline diff:
From 9a1450ecbe291cfc92e97f49506db9f443c8cbe9 Mon Sep 17 00:00:00 2001
From: Michael Paquier <[email protected]>
Date: Thu, 7 Jul 2022 16:51:18 +0900
Subject: [PATCH v2] Fix huge_pages on current Windows.
Since Windows 10 1703, it's necessary to pass a flag to MapViewOfFile()
to enable large pages at map time.
Reported-by: Okano Naoki
Author: Thomas Munro
Discussion: https://postgr.es/m/17448-0a96583a67edb1f7%40postgresql.org
---
src/backend/port/win32_shmem.c | 8 +++++++-
1 file changed, 7 insertions(+), 1 deletion(-)
diff --git a/src/backend/port/win32_shmem.c b/src/backend/port/win32_shmem.c
index 6cf69411db..2f51da5a6a 100644
--- a/src/backend/port/win32_shmem.c
+++ b/src/backend/port/win32_shmem.c
@@ -216,6 +216,7 @@ PGSharedMemoryCreate(Size size,
SIZE_T largePageSize = 0;
Size orig_size = size;
DWORD flProtect = PAGE_READWRITE;
+ DWORD desiredAccess;
ShmemProtectiveRegion = VirtualAlloc(NULL, PROTECTIVE_REGION_SIZE,
MEM_RESERVE, PAGE_NOACCESS);
@@ -353,12 +354,17 @@ retry:
if (!CloseHandle(hmap))
elog(LOG, "could not close handle to shared memory: error code %lu", GetLastError());
+ desiredAccess = FILE_MAP_WRITE | FILE_MAP_READ;
+
+ /* Set large pages if wanted. */
+ if ((flProtect & SEC_LARGE_PAGES) != 0)
+ desiredAccess |= FILE_MAP_LARGE_PAGES;
/*
* Get a pointer to the new shared memory segment. Map the whole segment
* at once, and let the system decide on the initial address.
*/
- memAddress = MapViewOfFileEx(hmap2, FILE_MAP_WRITE | FILE_MAP_READ, 0, 0, 0, NULL);
+ memAddress = MapViewOfFileEx(hmap2, desiredAccess, 0, 0, 0, NULL);
if (!memAddress)
ereport(FATAL,
(errmsg("could not create shared memory segment: error code %lu", GetLastError()),
--
2.36.1
[application/pgp-signature] signature.asc (833B, ../../[email protected]/3-signature.asc)
download
^ permalink raw reply [nested|flat] 22+ messages in thread
* Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work.
2022-03-25 07:52 BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. PG Bug reporting form <[email protected]>
2022-03-26 05:46 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Michael Paquier <[email protected]>
2022-03-26 08:24 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Julien Rouhaud <[email protected]>
2022-03-26 11:07 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Thomas Munro <[email protected]>
2022-03-30 07:54 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Michael Paquier <[email protected]>
2022-03-31 04:59 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Julien Rouhaud <[email protected]>
2022-03-31 07:42 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Michael Paquier <[email protected]>
2022-03-31 08:00 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Julien Rouhaud <[email protected]>
2022-03-31 10:46 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Julien Rouhaud <[email protected]>
@ 2022-04-01 05:54 ` Michael Paquier <[email protected]>
1 sibling, 0 replies; 22+ messages in thread
From: Michael Paquier @ 2022-04-01 05:54 UTC (permalink / raw)
To: Julien Rouhaud <[email protected]>; +Cc: Thomas Munro <[email protected]>; [email protected]; PostgreSQL mailing lists <[email protected]>
On Thu, Mar 31, 2022 at 06:46:59PM +0800, Julien Rouhaud wrote:
> So, after more digging it turns out that the patch is supposed to work. If I
> force using the PG_FILE_MAP_LARGE_PAGES, postgres starts and I do see "Locked
> WS" usage with VMMap, with a size in the order of magnitude of my
> shared_buffers.
>
> What is apparently not working on my VM is IsWindowsVersionOrGreater(10, 0,
> 1703). I added some debug around to check what GetVersionEx() [2] is saying,
> and I get:
>
> dwMajorVersion == 6
> dwMinorVersion == 2
> dwBuildNumber == 9200
Okay. Well, I'd like to think that the patch written as-is is
correct. Now your tests are saying the contrary, so I don't really
know what to think about it :)
>> TRUE if the specified version matches, or is greater than, the version of the
>> current Windows OS; otherwise, FALSE.
>
> Isn't that supposed to be the opposite?
I get from the upstream docs that if the runtime version of Windows is
higher than 10.0.1703, IsWindowsVersionOrGreater() should return
true. Perhaps the issue is in the patch and its argument values, but
it does not look straight-forward to know what those values should
be, and there are no examples in the docs to show that, either :/
--
Michael
Attachments:
[application/pgp-signature] signature.asc (833B, ../../[email protected]/2-signature.asc)
download
^ permalink raw reply [nested|flat] 22+ messages in thread
* Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work.
2022-03-25 07:52 BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. PG Bug reporting form <[email protected]>
2022-03-26 05:46 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Michael Paquier <[email protected]>
2022-03-26 08:24 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Julien Rouhaud <[email protected]>
2022-03-26 11:07 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Thomas Munro <[email protected]>
@ 2022-09-16 12:51 ` Michael Paquier <[email protected]>
2022-09-16 14:29 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Tom Lane <[email protected]>
2 siblings, 1 reply; 22+ messages in thread
From: Michael Paquier @ 2022-09-16 12:51 UTC (permalink / raw)
To: Thomas Munro <[email protected]>; +Cc: Julien Rouhaud <[email protected]>; [email protected]; PostgreSQL mailing lists <[email protected]>
On Sun, Mar 27, 2022 at 12:07:57AM +1300, Thomas Munro wrote:
> Some question I have: is FILE_MAP_LARGE PAGES a macro? We claim to
> support all those ancient zombie OSes like Windows 7, or maybe it's
> even XP for 11, and this has to be back-patched to 11, so we might
> need to make it conditional. But conditional on what? For example,
> does something like the attached work (untested)? What happens if a <
> 1703 kernel sees this flag, does it reject it or ignore it?
I have been looking at this thread, and found an answer in an example
of application creating a map object with large pages in [1]:
"This flag is ignored on OS versions before Windows 10, version 1703."
So based on that I think that we could just apply and backpatch what
you have here. This issue is much easier to reason about on HEAD
where we just care about Win >= 10, and we've be rather careful with
changes like that when it came to Windows. Any objections about doing
a backpatch? I'd like to do so after an extra lookup, if there are no
objections. Or would folks prefer a HEAD-only fix for now?
[1]: https://docs.microsoft.com/en-us/windows/win32/memory/creating-a-file-mapping-using-large-pages?sour...
--
Michael
Attachments:
[application/pgp-signature] signature.asc (833B, ../../[email protected]/2-signature.asc)
download
^ permalink raw reply [nested|flat] 22+ messages in thread
* Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work.
2022-03-25 07:52 BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. PG Bug reporting form <[email protected]>
2022-03-26 05:46 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Michael Paquier <[email protected]>
2022-03-26 08:24 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Julien Rouhaud <[email protected]>
2022-03-26 11:07 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Thomas Munro <[email protected]>
2022-09-16 12:51 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Michael Paquier <[email protected]>
@ 2022-09-16 14:29 ` Tom Lane <[email protected]>
2022-09-16 14:36 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Julien Rouhaud <[email protected]>
0 siblings, 1 reply; 22+ messages in thread
From: Tom Lane @ 2022-09-16 14:29 UTC (permalink / raw)
To: Michael Paquier <[email protected]>; +Cc: Thomas Munro <[email protected]>; Julien Rouhaud <[email protected]>; [email protected]; PostgreSQL mailing lists <[email protected]>
Michael Paquier <[email protected]> writes:
> So based on that I think that we could just apply and backpatch what
> you have here. This issue is much easier to reason about on HEAD
> where we just care about Win >= 10, and we've be rather careful with
> changes like that when it came to Windows. Any objections about doing
> a backpatch? I'd like to do so after an extra lookup, if there are no
> objections. Or would folks prefer a HEAD-only fix for now?
Let's just fix it in HEAD. I think the risk/reward ratio isn't very
good here.
(I'd be particularly against changing this in v10, because 10.23 will
be the last one; there will be no second chance if we ship it broken.)
regards, tom lane
^ permalink raw reply [nested|flat] 22+ messages in thread
* Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work.
2022-03-25 07:52 BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. PG Bug reporting form <[email protected]>
2022-03-26 05:46 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Michael Paquier <[email protected]>
2022-03-26 08:24 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Julien Rouhaud <[email protected]>
2022-03-26 11:07 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Thomas Munro <[email protected]>
2022-09-16 12:51 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Michael Paquier <[email protected]>
2022-09-16 14:29 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Tom Lane <[email protected]>
@ 2022-09-16 14:36 ` Julien Rouhaud <[email protected]>
2022-09-17 06:41 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Michael Paquier <[email protected]>
0 siblings, 1 reply; 22+ messages in thread
From: Julien Rouhaud @ 2022-09-16 14:36 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Michael Paquier <[email protected]>; Thomas Munro <[email protected]>; [email protected]; PostgreSQL mailing lists <[email protected]>
On Fri, Sep 16, 2022 at 10:29:38AM -0400, Tom Lane wrote:
> Michael Paquier <[email protected]> writes:
> > So based on that I think that we could just apply and backpatch what
> > you have here. This issue is much easier to reason about on HEAD
> > where we just care about Win >= 10, and we've be rather careful with
> > changes like that when it came to Windows. Any objections about doing
> > a backpatch? I'd like to do so after an extra lookup, if there are no
> > objections. Or would folks prefer a HEAD-only fix for now?
>
> Let's just fix it in HEAD. I think the risk/reward ratio isn't very
> good here.
>
> (I'd be particularly against changing this in v10, because 10.23 will
> be the last one; there will be no second chance if we ship it broken.)
+1
^ permalink raw reply [nested|flat] 22+ messages in thread
* Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work.
2022-03-25 07:52 BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. PG Bug reporting form <[email protected]>
2022-03-26 05:46 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Michael Paquier <[email protected]>
2022-03-26 08:24 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Julien Rouhaud <[email protected]>
2022-03-26 11:07 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Thomas Munro <[email protected]>
2022-09-16 12:51 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Michael Paquier <[email protected]>
2022-09-16 14:29 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Tom Lane <[email protected]>
2022-09-16 14:36 ` Re: BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. Julien Rouhaud <[email protected]>
@ 2022-09-17 06:41 ` Michael Paquier <[email protected]>
0 siblings, 0 replies; 22+ messages in thread
From: Michael Paquier @ 2022-09-17 06:41 UTC (permalink / raw)
To: Julien Rouhaud <[email protected]>; +Cc: Tom Lane <[email protected]>; Thomas Munro <[email protected]>; [email protected]; PostgreSQL mailing lists <[email protected]>
On Fri, Sep 16, 2022 at 10:36:12PM +0800, Julien Rouhaud wrote:
> On Fri, Sep 16, 2022 at 10:29:38AM -0400, Tom Lane wrote:
>> Let's just fix it in HEAD. I think the risk/reward ratio isn't very
>> good here.
>>
>> (I'd be particularly against changing this in v10, because 10.23 will
>> be the last one; there will be no second chance if we ship it broken.)
>
> +1
Okay, fine by me. I have applied that only on HEAD, then.
--
Michael
Attachments:
[application/pgp-signature] signature.asc (833B, ../../[email protected]/2-signature.asc)
download
^ permalink raw reply [nested|flat] 22+ messages in thread
end of thread, other threads:[~2022-09-17 06:41 UTC | newest]
Thread overview: 22+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2022-03-25 07:52 BUG #17448: In Windows 10, version 1703 and later, huge_pages doesn't work. PG Bug reporting form <[email protected]>
2022-03-26 05:46 ` Michael Paquier <[email protected]>
2022-03-26 08:24 ` Julien Rouhaud <[email protected]>
2022-03-26 09:03 ` Michael Paquier <[email protected]>
2022-03-26 11:07 ` Thomas Munro <[email protected]>
2022-03-26 11:33 ` Michael Paquier <[email protected]>
2022-03-30 07:54 ` Michael Paquier <[email protected]>
2022-03-31 04:59 ` Julien Rouhaud <[email protected]>
2022-03-31 07:42 ` Michael Paquier <[email protected]>
2022-03-31 08:00 ` Julien Rouhaud <[email protected]>
2022-03-31 10:46 ` Julien Rouhaud <[email protected]>
2022-03-31 11:03 ` Julien Rouhaud <[email protected]>
2022-04-01 05:59 ` Michael Paquier <[email protected]>
2022-04-01 06:19 ` Julien Rouhaud <[email protected]>
2022-04-26 04:54 ` Julien Rouhaud <[email protected]>
2022-04-27 08:13 ` Michael Paquier <[email protected]>
2022-07-07 22:38 ` Michael Paquier <[email protected]>
2022-04-01 05:54 ` Michael Paquier <[email protected]>
2022-09-16 12:51 ` Michael Paquier <[email protected]>
2022-09-16 14:29 ` Tom Lane <[email protected]>
2022-09-16 14:36 ` Julien Rouhaud <[email protected]>
2022-09-17 06:41 ` Michael Paquier <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox