public inbox for [email protected]
help / color / mirror / Atom feed[PATCH] Add native windows on arm64 support
12+ messages / 8 participants
[nested] [flat]
* [PATCH] Add native windows on arm64 support
@ 2022-02-23 10:09 Niyas Sait <[email protected]>
2022-03-18 20:50 ` Re: [PATCH] Add native windows on arm64 support Thomas Munro <[email protected]>
0 siblings, 1 reply; 12+ messages in thread
From: Niyas Sait @ 2022-02-23 10:09 UTC (permalink / raw)
To: [email protected]
Hello,
I have created a patch that adds support for building Postgres for the
windows/arm64 platform using the MSVC toolchain. Following changes have
been included
1. Extend MSVC scripts to handle ARM64 platform.
2. Add arm64 definition of spin_delay function.
3. Exclude arm_acle.h import with MSVC compiler.
Compilation steps are consistent and similar to other windows platforms.
The change has been tested on windows/x86_64 and windows/arm64 and all
regression tests passes on both platforms.
Thanks,
Niyas
Attachments:
[application/octet-stream] enable-native-windows-arm64-build.patch (5.7K, ../../CAFPTBD-74+AEuN9n7caJ0YUnW5A0r-KBX8rYoEJWqFPgLKpzdg@mail.gmail.com/3-enable-native-windows-arm64-build.patch)
download | inline diff:
From a83a393964736f5e73e0aff1eef446e52d4fc033 Mon Sep 17 00:00:00 2001
From: Niyas Sait <[email protected]>
Date: Tue, 22 Feb 2022 13:07:24 +0000
Subject: [PATCH] Enable postgres native build for windows-arm64 platform
Following changes are included
- Extend MSVC scripts to handle ARM64 platform
- Add arm64 definition of spin_delay function
- Exclude arm_acle.h import for MSVC
---
src/include/storage/s_lock.h | 10 +++++++++-
src/port/pg_crc32c_armv8.c | 2 ++
src/tools/msvc/MSBuildProject.pm | 15 +++++++++++----
src/tools/msvc/Mkvcbuild.pm | 9 +++++++--
src/tools/msvc/Solution.pm | 9 +++++++--
src/tools/msvc/gendef.pl | 8 ++++----
6 files changed, 40 insertions(+), 13 deletions(-)
diff --git a/src/include/storage/s_lock.h b/src/include/storage/s_lock.h
index 8a5a905e38..dfbbb5f10a 100644
--- a/src/include/storage/s_lock.h
+++ b/src/include/storage/s_lock.h
@@ -912,13 +912,21 @@ typedef LONG slock_t;
#define SPIN_DELAY() spin_delay()
/* If using Visual C++ on Win64, inline assembly is unavailable.
- * Use a _mm_pause intrinsic instead of rep nop.
+ * Use _mm_pause (x64) or __isb(arm64) intrinsic instead of rep nop.
*/
#if defined(_WIN64)
static __forceinline void
spin_delay(void)
{
+#ifdef _M_ARM64
+ /*
+ * arm64 way of hinting processor for spin loops optimisations
+ * ref: https://community.arm.com/support-forums/f/infrastructure-solutions-forum/48654/ssetoneon-faq
+ */
+ __isb(_ARM64_BARRIER_SY);
+#else
_mm_pause();
+#endif
}
#else
static __forceinline void
diff --git a/src/port/pg_crc32c_armv8.c b/src/port/pg_crc32c_armv8.c
index 9e301f96f6..981718752f 100644
--- a/src/port/pg_crc32c_armv8.c
+++ b/src/port/pg_crc32c_armv8.c
@@ -14,7 +14,9 @@
*/
#include "c.h"
+#ifndef _MSC_VER
#include <arm_acle.h>
+#endif
#include "port/pg_crc32c.h"
diff --git a/src/tools/msvc/MSBuildProject.pm b/src/tools/msvc/MSBuildProject.pm
index 5e312d232e..044907ef99 100644
--- a/src/tools/msvc/MSBuildProject.pm
+++ b/src/tools/msvc/MSBuildProject.pm
@@ -310,11 +310,18 @@ sub WriteItemDefinitionGroup
: ($self->{type} eq "dll" ? 'DynamicLibrary' : 'StaticLibrary');
my $libs = $self->GetAdditionalLinkerDependencies($cfgname, ';');
- my $targetmachine =
- $self->{platform} eq 'Win32' ? 'MachineX86' : 'MachineX64';
+ my $targetmachine;
+ if ($self->{platform} eq 'Win32') {
+ $targetmachine = 'MachineX86';
+ } elsif ($self->{platform} eq 'ARM64'){
+ $targetmachine = 'MachineARM64';
+ } else {
+ $targetmachine = 'MachineX64';
+ }
my $includes = join ';', @{$self->{includes}}, "";
-
+ # arm64 linker only supports dynamic base address
+ my $cfgrandbaseaddress = $self->{platform} eq 'ARM64' ? 'true' : 'false';
print $f <<EOF;
<ItemDefinitionGroup Condition="'\$(Configuration)|\$(Platform)'=='$cfgname|$self->{platform}'">
<ClCompile>
@@ -347,7 +354,7 @@ sub WriteItemDefinitionGroup
<ProgramDatabaseFile>.\\$cfgname\\$self->{name}\\$self->{name}.pdb</ProgramDatabaseFile>
<GenerateMapFile>false</GenerateMapFile>
<MapFileName>.\\$cfgname\\$self->{name}\\$self->{name}.map</MapFileName>
- <RandomizedBaseAddress>false</RandomizedBaseAddress>
+ <RandomizedBaseAddress>$cfgrandbaseaddress</RandomizedBaseAddress>
<!-- Permit links to MinGW-built, 32-bit DLLs (default before VS2012). -->
<ImageHasSafeExceptionHandlers/>
<SubSystem>Console</SubSystem>
diff --git a/src/tools/msvc/Mkvcbuild.pm b/src/tools/msvc/Mkvcbuild.pm
index 105f5c72a2..2ebc71b621 100644
--- a/src/tools/msvc/Mkvcbuild.pm
+++ b/src/tools/msvc/Mkvcbuild.pm
@@ -114,8 +114,13 @@ sub mkvcbuild
if ($vsVersion >= '9.00')
{
- push(@pgportfiles, 'pg_crc32c_sse42_choose.c');
- push(@pgportfiles, 'pg_crc32c_sse42.c');
+ if ($solution->{platform} eq 'ARM64') {
+ push(@pgportfiles, 'pg_crc32c_armv8_choose.c');
+ push(@pgportfiles, 'pg_crc32c_armv8.c');
+ } else {
+ push(@pgportfiles, 'pg_crc32c_sse42_choose.c');
+ push(@pgportfiles, 'pg_crc32c_sse42.c');
+ }
push(@pgportfiles, 'pg_crc32c_sb8.c');
}
else
diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm
index a21ea9bef9..106c0916ed 100644
--- a/src/tools/msvc/Solution.pm
+++ b/src/tools/msvc/Solution.pm
@@ -67,8 +67,13 @@ sub DeterminePlatform
# Examine CL help output to determine if we are in 32 or 64-bit mode.
my $output = `cl /help 2>&1`;
$? >> 8 == 0 or die "cl command not found";
- $self->{platform} =
- ($output =~ /^\/favor:<.+AMD64/m) ? 'x64' : 'Win32';
+ if ($output =~ /^\/favor:<.+AMD64/m) {
+ $self->{platform} = 'x64';
+ } elsif($output =~ /for ARM64$/m) {
+ $self->{platform} = 'ARM64';
+ } else {
+ $self->{platform} = 'Win32';
+ }
}
else
{
diff --git a/src/tools/msvc/gendef.pl b/src/tools/msvc/gendef.pl
index b8c514a831..2068484b14 100644
--- a/src/tools/msvc/gendef.pl
+++ b/src/tools/msvc/gendef.pl
@@ -120,9 +120,9 @@ sub writedef
{
my $isdata = $def->{$f} eq 'data';
- # Strip the leading underscore for win32, but not x64
+ # Strip the leading underscore for win32, but not x64 and arm64
$f =~ s/^_//
- unless ($platform eq "x64");
+ unless ($platform ne "Win32");
# Emit just the name if it's a function symbol, or emit the name
# decorated with the DATA option for variables.
@@ -144,13 +144,13 @@ sub usage
{
die( "Usage: gendef.pl <modulepath> <platform>\n"
. " modulepath: path to dir with obj files, no trailing slash"
- . " platform: Win32 | x64");
+ . " platform: Win32 | x64 | ARM64");
}
usage()
unless scalar(@ARGV) == 2
&& ( ($ARGV[0] =~ /\\([^\\]+$)/)
- && ($ARGV[1] eq 'Win32' || $ARGV[1] eq 'x64'));
+ && ($ARGV[1] eq 'Win32' || $ARGV[1] eq 'x64' || $ARGV[1] eq 'ARM64'));
my $defname = uc $1;
my $deffile = "$ARGV[0]/$defname.def";
my $platform = $ARGV[1];
--
2.35.0.windows.1
^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: [PATCH] Add native windows on arm64 support
2022-02-23 10:09 [PATCH] Add native windows on arm64 support Niyas Sait <[email protected]>
@ 2022-03-18 20:50 ` Thomas Munro <[email protected]>
2022-03-22 09:37 ` Re: [PATCH] Add native windows on arm64 support Niyas Sait <[email protected]>
0 siblings, 1 reply; 12+ messages in thread
From: Thomas Munro @ 2022-03-18 20:50 UTC (permalink / raw)
To: Niyas Sait <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]>
On Wed, Feb 23, 2022 at 11:09 PM Niyas Sait <[email protected]> wrote:
> I have created a patch that adds support for building Postgres for the windows/arm64 platform using the MSVC toolchain. Following changes have been included
>
> 1. Extend MSVC scripts to handle ARM64 platform.
> 2. Add arm64 definition of spin_delay function.
> 3. Exclude arm_acle.h import with MSVC compiler.
>
> Compilation steps are consistent and similar to other windows platforms.
>
> The change has been tested on windows/x86_64 and windows/arm64 and all regression tests passes on both platforms.
+ # arm64 linker only supports dynamic base address
+ my $cfgrandbaseaddress = $self->{platform} eq 'ARM64' ? 'true' : 'false';
...
+ <RandomizedBaseAddress>$cfgrandbaseaddress</RandomizedBaseAddress>
Does that mean that you can't turn off ASLR on this arch? Does it
cause random backend failures due to inability to map shared memory at
the expected address? Our experience with EXEC_BACKEND on various
Unix systems tells us that you have to turn off ASLR if you want 100%
success rate on starting backends, but I guess it depends on the
details of how the randomisation is done.
Any interest in providing a build farm animal, so that we could know
that this works?
^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: [PATCH] Add native windows on arm64 support
2022-02-23 10:09 [PATCH] Add native windows on arm64 support Niyas Sait <[email protected]>
2022-03-18 20:50 ` Re: [PATCH] Add native windows on arm64 support Thomas Munro <[email protected]>
@ 2022-03-22 09:37 ` Niyas Sait <[email protected]>
2022-03-22 10:30 ` Re: [PATCH] Add native windows on arm64 support Julien Rouhaud <[email protected]>
0 siblings, 1 reply; 12+ messages in thread
From: Niyas Sait @ 2022-03-22 09:37 UTC (permalink / raw)
To: Thomas Munro <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]>
Thanks, Thomas for reviewing the patch!
Yes, we cannot turn off ASLR for Arm64 targets with MSVC. I haven't seen
any issues so far on win/arm64 with this option turned off. I guess it
should be okay. If we really need ASLR turned off, then I guess might have
to use clang.
Yes, we could look into providing a build machine. Do you have any
reference to what the CI system looks like now for PostgresSQL and how to
add new workers etc.?
Niyas
On Fri, 18 Mar 2022 at 20:50, Thomas Munro <[email protected]> wrote:
> On Wed, Feb 23, 2022 at 11:09 PM Niyas Sait <[email protected]> wrote:
> > I have created a patch that adds support for building Postgres for the
> windows/arm64 platform using the MSVC toolchain. Following changes have
> been included
> >
> > 1. Extend MSVC scripts to handle ARM64 platform.
> > 2. Add arm64 definition of spin_delay function.
> > 3. Exclude arm_acle.h import with MSVC compiler.
> >
> > Compilation steps are consistent and similar to other windows platforms.
> >
> > The change has been tested on windows/x86_64 and windows/arm64 and all
> regression tests passes on both platforms.
>
> + # arm64 linker only supports dynamic base address
> + my $cfgrandbaseaddress = $self->{platform} eq 'ARM64' ? 'true' :
> 'false';
> ...
> + <RandomizedBaseAddress>$cfgrandbaseaddress</RandomizedBaseAddress>
>
> Does that mean that you can't turn off ASLR on this arch? Does it
> cause random backend failures due to inability to map shared memory at
> the expected address? Our experience with EXEC_BACKEND on various
> Unix systems tells us that you have to turn off ASLR if you want 100%
> success rate on starting backends, but I guess it depends on the
> details of how the randomisation is done.
>
> Any interest in providing a build farm animal, so that we could know
> that this works?
>
^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: [PATCH] Add native windows on arm64 support
2022-02-23 10:09 [PATCH] Add native windows on arm64 support Niyas Sait <[email protected]>
2022-03-18 20:50 ` Re: [PATCH] Add native windows on arm64 support Thomas Munro <[email protected]>
2022-03-22 09:37 ` Re: [PATCH] Add native windows on arm64 support Niyas Sait <[email protected]>
@ 2022-03-22 10:30 ` Julien Rouhaud <[email protected]>
2022-03-23 03:30 ` Re: [PATCH] Add native windows on arm64 support Thomas Munro <[email protected]>
0 siblings, 1 reply; 12+ messages in thread
From: Julien Rouhaud @ 2022-03-22 10:30 UTC (permalink / raw)
To: Niyas Sait <[email protected]>; +Cc: Thomas Munro <[email protected]>; PostgreSQL Hackers <[email protected]>
Hi,
Please don't top-post here. See
https://wiki.postgresql.org/wiki/Mailing_Lists#Email_etiquette_mechanics.
On Tue, Mar 22, 2022 at 09:37:46AM +0000, Niyas Sait wrote:
>
> Yes, we could look into providing a build machine. Do you have any
> reference to what the CI system looks like now for PostgresSQL and how to
> add new workers etc.?
It's all documented at
https://wiki.postgresql.org/wiki/PostgreSQL_Buildfarm_Howto.
^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: [PATCH] Add native windows on arm64 support
2022-02-23 10:09 [PATCH] Add native windows on arm64 support Niyas Sait <[email protected]>
2022-03-18 20:50 ` Re: [PATCH] Add native windows on arm64 support Thomas Munro <[email protected]>
2022-03-22 09:37 ` Re: [PATCH] Add native windows on arm64 support Niyas Sait <[email protected]>
2022-03-22 10:30 ` Re: [PATCH] Add native windows on arm64 support Julien Rouhaud <[email protected]>
@ 2022-03-23 03:30 ` Thomas Munro <[email protected]>
2022-03-24 01:15 ` Re: [PATCH] Add native windows on arm64 support Andres Freund <[email protected]>
0 siblings, 1 reply; 12+ messages in thread
From: Thomas Munro @ 2022-03-23 03:30 UTC (permalink / raw)
To: Julien Rouhaud <[email protected]>; +Cc: Niyas Sait <[email protected]>; PostgreSQL Hackers <[email protected]>
On Tue, Mar 22, 2022 at 11:30 PM Julien Rouhaud <[email protected]> wrote:
> On Tue, Mar 22, 2022 at 09:37:46AM +0000, Niyas Sait wrote:
> > Yes, we could look into providing a build machine. Do you have any
> > reference to what the CI system looks like now for PostgresSQL and how to
> > add new workers etc.?
>
> It's all documented at
> https://wiki.postgresql.org/wiki/PostgreSQL_Buildfarm_Howto.
It seems likely there will be more and more Windows/ARM users, so yeah
having a machine to test that combination would be great. I wonder if
ASLR isn't breaking for you by good luck only...
^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: [PATCH] Add native windows on arm64 support
2022-02-23 10:09 [PATCH] Add native windows on arm64 support Niyas Sait <[email protected]>
2022-03-18 20:50 ` Re: [PATCH] Add native windows on arm64 support Thomas Munro <[email protected]>
2022-03-22 09:37 ` Re: [PATCH] Add native windows on arm64 support Niyas Sait <[email protected]>
2022-03-22 10:30 ` Re: [PATCH] Add native windows on arm64 support Julien Rouhaud <[email protected]>
2022-03-23 03:30 ` Re: [PATCH] Add native windows on arm64 support Thomas Munro <[email protected]>
@ 2022-03-24 01:15 ` Andres Freund <[email protected]>
2022-03-24 22:38 ` Re: [PATCH] Add native windows on arm64 support Thomas Munro <[email protected]>
0 siblings, 1 reply; 12+ messages in thread
From: Andres Freund @ 2022-03-24 01:15 UTC (permalink / raw)
To: Thomas Munro <[email protected]>; +Cc: Julien Rouhaud <[email protected]>; Niyas Sait <[email protected]>; PostgreSQL Hackers <[email protected]>
Hi,
On 2022-03-23 16:30:58 +1300, Thomas Munro wrote:
> On Tue, Mar 22, 2022 at 11:30 PM Julien Rouhaud <[email protected]> wrote:
> > On Tue, Mar 22, 2022 at 09:37:46AM +0000, Niyas Sait wrote:
> > > Yes, we could look into providing a build machine. Do you have any
> > > reference to what the CI system looks like now for PostgresSQL and how to
> > > add new workers etc.?
> >
> > It's all documented at
> > https://wiki.postgresql.org/wiki/PostgreSQL_Buildfarm_Howto.
>
> It seems likely there will be more and more Windows/ARM users, so yeah
> having a machine to test that combination would be great. I wonder if
> ASLR isn't breaking for you by good luck only...
I think we've generally seen the ASLR issue become less prominent on
windows. Whether that's because of the silent retries we added, or because
just about everyone moved to 64bit windows / PG, I don't know. I'd guess both,
with 64bit being the larger influence.
Wonder if it's worth adding some debug logging to the retry code and stop
disabling ASLR on 64bit windows... It's imo pretty crazy that we loop up to
100 times in internal_forkexec() around CreateProcess() &&
pgwin32_ReserveSharedMemoryRegion() without, as far as I can see, a single
debug message.
I don't think we can infer too much about the failure rate on windows from the
!windows EXEC_BACKEND rates. The two internal_forkexec() implementations
behaves just too differently.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: [PATCH] Add native windows on arm64 support
2022-02-23 10:09 [PATCH] Add native windows on arm64 support Niyas Sait <[email protected]>
2022-03-18 20:50 ` Re: [PATCH] Add native windows on arm64 support Thomas Munro <[email protected]>
2022-03-22 09:37 ` Re: [PATCH] Add native windows on arm64 support Niyas Sait <[email protected]>
2022-03-22 10:30 ` Re: [PATCH] Add native windows on arm64 support Julien Rouhaud <[email protected]>
2022-03-23 03:30 ` Re: [PATCH] Add native windows on arm64 support Thomas Munro <[email protected]>
2022-03-24 01:15 ` Re: [PATCH] Add native windows on arm64 support Andres Freund <[email protected]>
@ 2022-03-24 22:38 ` Thomas Munro <[email protected]>
2022-04-07 06:36 ` Re: [PATCH] Add native windows on arm64 support Michael Paquier <[email protected]>
0 siblings, 1 reply; 12+ messages in thread
From: Thomas Munro @ 2022-03-24 22:38 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Julien Rouhaud <[email protected]>; Niyas Sait <[email protected]>; PostgreSQL Hackers <[email protected]>
On Thu, Mar 24, 2022 at 2:15 PM Andres Freund <[email protected]> wrote:
> I think we've generally seen the ASLR issue become less prominent on
> windows. Whether that's because of the silent retries we added, or because
> just about everyone moved to 64bit windows / PG, I don't know. I'd guess both,
> with 64bit being the larger influence.
>
> Wonder if it's worth adding some debug logging to the retry code and stop
> disabling ASLR on 64bit windows... It's imo pretty crazy that we loop up to
> 100 times in internal_forkexec() around CreateProcess() &&
> pgwin32_ReserveSharedMemoryRegion() without, as far as I can see, a single
> debug message.
Yeah. I think we should commit this patch, but decree that
Windows/aarch64 support is experimental only for now. That allows a
build farm animal to be set up. Then we add a bit of extra logging
and see how it does running our test suite over time and learn more.
^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: [PATCH] Add native windows on arm64 support
2022-02-23 10:09 [PATCH] Add native windows on arm64 support Niyas Sait <[email protected]>
2022-03-18 20:50 ` Re: [PATCH] Add native windows on arm64 support Thomas Munro <[email protected]>
2022-03-22 09:37 ` Re: [PATCH] Add native windows on arm64 support Niyas Sait <[email protected]>
2022-03-22 10:30 ` Re: [PATCH] Add native windows on arm64 support Julien Rouhaud <[email protected]>
2022-03-23 03:30 ` Re: [PATCH] Add native windows on arm64 support Thomas Munro <[email protected]>
2022-03-24 01:15 ` Re: [PATCH] Add native windows on arm64 support Andres Freund <[email protected]>
2022-03-24 22:38 ` Re: [PATCH] Add native windows on arm64 support Thomas Munro <[email protected]>
@ 2022-04-07 06:36 ` Michael Paquier <[email protected]>
2022-04-07 13:40 ` Re: [PATCH] Add native windows on arm64 support Tom Lane <[email protected]>
0 siblings, 1 reply; 12+ messages in thread
From: Michael Paquier @ 2022-04-07 06:36 UTC (permalink / raw)
To: Thomas Munro <[email protected]>; +Cc: Andres Freund <[email protected]>; Julien Rouhaud <[email protected]>; Niyas Sait <[email protected]>; PostgreSQL Hackers <[email protected]>
On Fri, Mar 25, 2022 at 11:38:44AM +1300, Thomas Munro wrote:
> Yeah. I think we should commit this patch, but decree that
> Windows/aarch64 support is experimental only for now. That allows a
> build farm animal to be set up. Then we add a bit of extra logging
> and see how it does running our test suite over time and learn more.
I don't have such a setup so my testing capabilities are limited.
Does anybody have one? I think that we could be flexible for this
patch, even after feature freeze as it introduces something entirely
new without impacting the existing code. The patch has been moved to
the next CF for now.
--
Michael
Attachments:
[application/pgp-signature] signature.asc (833B, ../../[email protected]/2-signature.asc)
download
^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: [PATCH] Add native windows on arm64 support
2022-02-23 10:09 [PATCH] Add native windows on arm64 support Niyas Sait <[email protected]>
2022-03-18 20:50 ` Re: [PATCH] Add native windows on arm64 support Thomas Munro <[email protected]>
2022-03-22 09:37 ` Re: [PATCH] Add native windows on arm64 support Niyas Sait <[email protected]>
2022-03-22 10:30 ` Re: [PATCH] Add native windows on arm64 support Julien Rouhaud <[email protected]>
2022-03-23 03:30 ` Re: [PATCH] Add native windows on arm64 support Thomas Munro <[email protected]>
2022-03-24 01:15 ` Re: [PATCH] Add native windows on arm64 support Andres Freund <[email protected]>
2022-03-24 22:38 ` Re: [PATCH] Add native windows on arm64 support Thomas Munro <[email protected]>
2022-04-07 06:36 ` Re: [PATCH] Add native windows on arm64 support Michael Paquier <[email protected]>
@ 2022-04-07 13:40 ` Tom Lane <[email protected]>
2022-04-07 14:55 ` Re: [PATCH] Add native windows on arm64 support Daniel Gustafsson <[email protected]>
2022-04-07 17:22 ` Re: [PATCH] Add native windows on arm64 support Andres Freund <[email protected]>
0 siblings, 2 replies; 12+ messages in thread
From: Tom Lane @ 2022-04-07 13:40 UTC (permalink / raw)
To: Michael Paquier <[email protected]>; +Cc: Thomas Munro <[email protected]>; Andres Freund <[email protected]>; Julien Rouhaud <[email protected]>; Niyas Sait <[email protected]>; PostgreSQL Hackers <[email protected]>
Michael Paquier <[email protected]> writes:
> On Fri, Mar 25, 2022 at 11:38:44AM +1300, Thomas Munro wrote:
>> Yeah. I think we should commit this patch, but decree that
>> Windows/aarch64 support is experimental only for now. That allows a
>> build farm animal to be set up. Then we add a bit of extra logging
>> and see how it does running our test suite over time and learn more.
> I don't have such a setup so my testing capabilities are limited.
> Does anybody have one? I think that we could be flexible for this
> patch, even after feature freeze as it introduces something entirely
> new without impacting the existing code. The patch has been moved to
> the next CF for now.
I dunno, the lack of any in-house capability for this makes me very
nervous. If it causes problems down the road, how will we debug it?
So it seems like the sort of patch to put in at the beginning of a
development cycle, not post-feature-freeze.
regards, tom lane
^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: [PATCH] Add native windows on arm64 support
2022-02-23 10:09 [PATCH] Add native windows on arm64 support Niyas Sait <[email protected]>
2022-03-18 20:50 ` Re: [PATCH] Add native windows on arm64 support Thomas Munro <[email protected]>
2022-03-22 09:37 ` Re: [PATCH] Add native windows on arm64 support Niyas Sait <[email protected]>
2022-03-22 10:30 ` Re: [PATCH] Add native windows on arm64 support Julien Rouhaud <[email protected]>
2022-03-23 03:30 ` Re: [PATCH] Add native windows on arm64 support Thomas Munro <[email protected]>
2022-03-24 01:15 ` Re: [PATCH] Add native windows on arm64 support Andres Freund <[email protected]>
2022-03-24 22:38 ` Re: [PATCH] Add native windows on arm64 support Thomas Munro <[email protected]>
2022-04-07 06:36 ` Re: [PATCH] Add native windows on arm64 support Michael Paquier <[email protected]>
2022-04-07 13:40 ` Re: [PATCH] Add native windows on arm64 support Tom Lane <[email protected]>
@ 2022-04-07 14:55 ` Daniel Gustafsson <[email protected]>
1 sibling, 0 replies; 12+ messages in thread
From: Daniel Gustafsson @ 2022-04-07 14:55 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Michael Paquier <[email protected]>; Thomas Munro <[email protected]>; Andres Freund <[email protected]>; Julien Rouhaud <[email protected]>; Niyas Sait <[email protected]>; PostgreSQL Hackers <[email protected]>
> On 7 Apr 2022, at 15:40, Tom Lane <[email protected]> wrote:
>
> Michael Paquier <[email protected]> writes:
>> On Fri, Mar 25, 2022 at 11:38:44AM +1300, Thomas Munro wrote:
>>> Yeah. I think we should commit this patch, but decree that
>>> Windows/aarch64 support is experimental only for now. That allows a
>>> build farm animal to be set up. Then we add a bit of extra logging
>>> and see how it does running our test suite over time and learn more.
>
>> I don't have such a setup so my testing capabilities are limited.
>> Does anybody have one? I think that we could be flexible for this
>> patch, even after feature freeze as it introduces something entirely
>> new without impacting the existing code. The patch has been moved to
>> the next CF for now.
>
> I dunno, the lack of any in-house capability for this makes me very
> nervous. If it causes problems down the road, how will we debug it?
If those with an interest in such platform support cannot spare the cycles to
at least run a buildfarm member, then it seems a stretch for us to maintain
such support with any confidence.
> So it seems like the sort of patch to put in at the beginning of a
> development cycle, not post-feature-freeze.
+1
--
Daniel Gustafsson https://vmware.com/
^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: [PATCH] Add native windows on arm64 support
2022-02-23 10:09 [PATCH] Add native windows on arm64 support Niyas Sait <[email protected]>
2022-03-18 20:50 ` Re: [PATCH] Add native windows on arm64 support Thomas Munro <[email protected]>
2022-03-22 09:37 ` Re: [PATCH] Add native windows on arm64 support Niyas Sait <[email protected]>
2022-03-22 10:30 ` Re: [PATCH] Add native windows on arm64 support Julien Rouhaud <[email protected]>
2022-03-23 03:30 ` Re: [PATCH] Add native windows on arm64 support Thomas Munro <[email protected]>
2022-03-24 01:15 ` Re: [PATCH] Add native windows on arm64 support Andres Freund <[email protected]>
2022-03-24 22:38 ` Re: [PATCH] Add native windows on arm64 support Thomas Munro <[email protected]>
2022-04-07 06:36 ` Re: [PATCH] Add native windows on arm64 support Michael Paquier <[email protected]>
2022-04-07 13:40 ` Re: [PATCH] Add native windows on arm64 support Tom Lane <[email protected]>
@ 2022-04-07 17:22 ` Andres Freund <[email protected]>
1 sibling, 0 replies; 12+ messages in thread
From: Andres Freund @ 2022-04-07 17:22 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Michael Paquier <[email protected]>; Thomas Munro <[email protected]>; Julien Rouhaud <[email protected]>; Niyas Sait <[email protected]>; PostgreSQL Hackers <[email protected]>
Hi,
On 2022-04-07 09:40:43 -0400, Tom Lane wrote:
> If it causes problems down the road, how will we debug it?
If what causes problems down the road? Afaics the patch doesn't change
anything outside of windows-on-arm, so it shouldn't cause any breakage we care
about until we get a buildfarm animal.
We've traditionally been somewhat relaxed about adding support for new
platforms, on similar notions. That said:
> So it seems like the sort of patch to put in at the beginning of a
> development cycle, not post-feature-freeze.
There doesn't seem to be a great urgency, and there's plenty stuff going on
right now. I can see us merging it post branching off, and then backpatching
it a bit later?
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 12+ messages in thread
* [PATCH v7 09/16] Make opp freeze heuristic compatible with prune+freeze record
@ 2024-03-26 00:48 Melanie Plageman <[email protected]>
0 siblings, 0 replies; 12+ messages in thread
From: Melanie Plageman @ 2024-03-26 00:48 UTC (permalink / raw)
Once the prune and freeze records are combined, we will no longer be
able to use a test of whether or not pruning emitted an FPI to decide
whether or not to opportunistically freeze a freezable page.
While this heuristic should be improved, for now, approximate the
previous logic by keeping track of whether or not a hint bit FPI was
emitted during visibility checks (when checksums are on) and combine
that with checking XLogCheckBufferNeedsBackup(). If we just finished
deciding whether or not to prune and the current buffer seems to need an
FPI after modification, it is likely that pruning would have emitted an
FPI.
---
src/backend/access/heap/pruneheap.c | 57 +++++++++++++++++++++--------
1 file changed, 42 insertions(+), 15 deletions(-)
diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
index e009c7579dd..d38de9b063d 100644
--- a/src/backend/access/heap/pruneheap.c
+++ b/src/backend/access/heap/pruneheap.c
@@ -247,6 +247,10 @@ heap_page_prune_and_freeze(Relation relation, Buffer buffer,
TransactionId visibility_cutoff_xid;
bool do_freeze;
bool all_visible_except_removable;
+ bool do_prune;
+ bool whole_page_freezable;
+ bool hint_bit_fpi;
+ bool prune_fpi = false;
int64 fpi_before = pgWalUsage.wal_fpi;
/*
@@ -456,6 +460,13 @@ heap_page_prune_and_freeze(Relation relation, Buffer buffer,
}
}
+ /*
+ * If checksums are enabled, heap_prune_satisfies_vacuum() may have caused
+ * an FPI to be emitted. Then reset fpi_before for no prune case.
+ */
+ hint_bit_fpi = fpi_before != pgWalUsage.wal_fpi;
+ fpi_before = pgWalUsage.wal_fpi;
+
/*
* For vacuum, if the whole page will become frozen, we consider
* opportunistically freezing tuples. Dead tuples which will be removed by
@@ -500,11 +511,41 @@ heap_page_prune_and_freeze(Relation relation, Buffer buffer,
if (off_loc)
*off_loc = InvalidOffsetNumber;
+ do_prune = prstate.nredirected > 0 ||
+ prstate.ndead > 0 ||
+ prstate.nunused > 0;
+
+ /*
+ * Only incur overhead of checking if we will do an FPI if we might use
+ * the information.
+ */
+ if (do_prune && pagefrz)
+ prune_fpi = XLogCheckBufferNeedsBackup(buffer);
+
+ /* Is the whole page freezable? And is there something to freeze */
+ whole_page_freezable = all_visible_except_removable &&
+ presult->all_frozen;
+
+ /*
+ * Freeze the page when heap_prepare_freeze_tuple indicates that at least
+ * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also
+ * freeze when pruning generated an FPI, if doing so means that we set the
+ * page all-frozen afterwards (might not happen until final heap pass).
+ * XXX: Previously, we knew if pruning emitted an FPI by checking
+ * pgWalUsage.wal_fpi before and after pruning. Once the freeze and prune
+ * records are combined, this heuristic couldn't be used anymore. The
+ * opportunistic freeze heuristic must be improved; however, for now, try
+ * to approximate it.
+ */
+ do_freeze = pagefrz &&
+ (pagefrz->freeze_required ||
+ (whole_page_freezable && presult->nfrozen > 0 && (prune_fpi || hint_bit_fpi)));
+
/* Any error while applying the changes is critical */
START_CRIT_SECTION();
/* Have we found any prunable items? */
- if (prstate.nredirected > 0 || prstate.ndead > 0 || prstate.nunused > 0)
+ if (do_prune)
{
/*
* Apply the planned item changes, then repair page fragmentation, and
@@ -569,20 +610,6 @@ heap_page_prune_and_freeze(Relation relation, Buffer buffer,
/* Record number of newly-set-LP_DEAD items for caller */
presult->nnewlpdead = prstate.ndead;
- /*
- * Freeze the page when heap_prepare_freeze_tuple indicates that at least
- * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also
- * freeze when pruning generated an FPI, if doing so means that we set the
- * page all-frozen afterwards (might not happen until final heap pass).
- */
- if (pagefrz)
- do_freeze = pagefrz->freeze_required ||
- (all_visible_except_removable && presult->all_frozen &&
- presult->nfrozen > 0 &&
- fpi_before != pgWalUsage.wal_fpi);
- else
- do_freeze = false;
-
if (do_freeze)
{
TransactionId frz_conflict_horizon = InvalidTransactionId;
--
2.40.1
--ck6erxojvlx23byk
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v7-0010-Separate-tuple-pre-freeze-checks-and-invoke-earli.patch"
^ permalink raw reply [nested|flat] 12+ messages in thread
end of thread, other threads:[~2024-03-26 00:48 UTC | newest]
Thread overview: 12+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2022-02-23 10:09 [PATCH] Add native windows on arm64 support Niyas Sait <[email protected]>
2022-03-18 20:50 ` Thomas Munro <[email protected]>
2022-03-22 09:37 ` Niyas Sait <[email protected]>
2022-03-22 10:30 ` Julien Rouhaud <[email protected]>
2022-03-23 03:30 ` Thomas Munro <[email protected]>
2022-03-24 01:15 ` Andres Freund <[email protected]>
2022-03-24 22:38 ` Thomas Munro <[email protected]>
2022-04-07 06:36 ` Michael Paquier <[email protected]>
2022-04-07 13:40 ` Tom Lane <[email protected]>
2022-04-07 14:55 ` Daniel Gustafsson <[email protected]>
2022-04-07 17:22 ` Andres Freund <[email protected]>
2024-03-26 00:48 [PATCH v7 09/16] Make opp freeze heuristic compatible with prune+freeze record Melanie Plageman <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox