Date: Thu, 28 Jul 2022 17:44:28 -0500
From: Justin Pryzby <pryzby@telsasoft.com>
To: Melih Mutlu <m.melihmutlu@gmail.com>
Cc: pgsql-hackers@postgresql.org, Andres Freund <andres@anarazel.de>,
	Andrew Dunstan <andrew@dunslane.net>,
	Thomas Munro <thomas.munro@gmail.com>
Subject: Re: Mingw task for Cirrus CI
Message-ID: <20220728224428.GN15006@telsasoft.com>
References: 
 <CAGPVpCSKS9E0An4=e7ZDnme+y=WOcQFJYJegKO8kE9=gh8NJKQ@mail.gmail.com>
 <20220322001559.mln5mexnhvijuntn@alap3.anarazel.de>
 <CAGPVpCRTg_zUfDB393EZLkuMkJ0UMu_N06TpgvAjH4Y8NcMzfg@mail.gmail.com>
 <20220331002618.npsfms3sijof7z7n@alap3.anarazel.de>
 <99db8656-9722-65c4-39b7-59fc14670c66@dunslane.net>
 <20220407171021.h5cnwfs355zdug7v@alap3.anarazel.de>
 <20220409002758.GM24419@telsasoft.com>
 <20220409010241.7httalsahzg7rmyl@alap3.anarazel.de>
 <e41337cb-a34d-df3a-5e82-fd7bc1e8080f@dunslane.net>
 <CAGPVpCTW1u8DGdwgSBsDyziE_7Tk7UwgOQ2tVR650F544sybDQ@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="Ca0e2zgpnh8/XhnM"
Content-Disposition: inline
In-Reply-To: 
 <CAGPVpCTW1u8DGdwgSBsDyziE_7Tk7UwgOQ2tVR650F544sybDQ@mail.gmail.com>
User-Agent: Mutt/1.9.4 (2018-02-28)
Archived-At: 
 <https://www.postgresql.org/message-id/20220728224428.GN15006%40telsasoft.com>
Precedence: bulk


--Ca0e2zgpnh8/XhnM
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline

I think the "only_if" should allow separately running one but not both of the
windows instances, like:

+  only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*mingw64'

I'm not sure, but maybe this task should only run "by request", and omit the
first condition:

+  only_if: $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*mingw64'

I think it should include something like

+  setup_additional_packages_script: |
+    REM C:\msys64\usr\bin\pacman.exe -S --noconfirm ...

Let's see what others think about those.

Do you know if this handles logging of crash dumps ?

With tweaks to prep_buildtree, and with ./configure --cache-file, that step
goes down to ~36sec (unless configure needs to be re-run).

I also looked into using busybox to avoid running separate processes for each
"ln", but I think 36sec is good enough.

At one point, I tried setting "CIRRUS_SHELL: bash" to avoid writing "bash -c"
over and over, but never got it working.

-- 
Justin

--Ca0e2zgpnh8/XhnM
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
 filename="0001-Added-Windows-with-MinGW-environment-in-Cirrus-CI.patch"

From ec28354e1e526ddab3a8df90540f9aa127c40193 Mon Sep 17 00:00:00 2001
From: Melih Mutlu <memutlu@microsoft.com>
Date: Mon, 21 Feb 2022 14:46:05 +0300
Subject: [PATCH 1/2] Added Windows with MinGW environment in Cirrus CI

---
 .cirrus.yml | 78 ++++++++++++++++++++++++++++++++++++++++++++---------
 1 file changed, 65 insertions(+), 13 deletions(-)

diff --git a/.cirrus.yml b/.cirrus.yml
index f23d6cae552..1a06cdcaadb 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -338,13 +338,29 @@ task:
     cores_script: src/tools/ci/cores_backtrace.sh macos "${HOME}/cores"
 
 
+WINDOWS_ENVIRONMENT_BASE: &WINDOWS_ENVIRONMENT_BASE
+    env:
+      # Half the allowed per-user CPU cores
+      CPUS: 4
+      # The default working dir is in a directory msbuild complains about
+      CIRRUS_WORKING_DIR: "c:/cirrus"
+
+      # Avoids port conflicts between concurrent tap test runs
+      PG_TEST_USE_UNIX_SOCKETS: 1
+
+    only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*windows.*'
+
+    sysinfo_script: |
+      chcp
+      systeminfo
+      powershell -Command get-psdrive -psprovider filesystem
+      set
+
 task:
+  << : *WINDOWS_ENVIRONMENT_BASE
   name: Windows - Server 2019, VS 2019
 
   env:
-    # Half the allowed per-user CPU cores
-    CPUS: 4
-
     # Our windows infrastructure doesn't have test concurrency above the level
     # of a single vcregress test target. Due to that, it's useful to run prove
     # with multiple jobs. For the other tasks it isn't, because two sources
@@ -354,15 +370,11 @@ task:
     # likely can be improved upon further.
     PROVE_FLAGS: -j10 --timer
 
-    # The default cirrus working dir is in a directory msbuild complains about
-    CIRRUS_WORKING_DIR: "c:/cirrus"
     # Avoid re-installing over and over
     NO_TEMP_INSTALL: 1
     # git's tar doesn't deal with drive letters, see
     # https://postgr.es/m/b6782dc3-a7b0-ed56-175f-f8f54cb08d67%40dunslane.net
     TAR: "c:/windows/system32/tar.exe"
-    # Avoids port conflicts between concurrent tap test runs
-    PG_TEST_USE_UNIX_SOCKETS: 1
     PG_REGRESS_SOCK_DIR: "c:/cirrus/"
     # -m enables parallelism
     # verbosity:minimal + Summary reduce verbosity, while keeping a summary of
@@ -393,12 +405,6 @@ task:
     cpu: $CPUS
     memory: 4G
 
-  sysinfo_script: |
-    chcp
-    systeminfo
-    powershell -Command get-psdrive -psprovider filesystem
-    set
-
   setup_additional_packages_script: |
     REM choco install -y --no-progress ...
 
@@ -456,6 +462,52 @@ task:
       path: "crashlog-*.txt"
       type: text/plain
 
+task:
+  << : *WINDOWS_ENVIRONMENT_BASE
+  name: Windows - Server 2019, MinGW64
+  windows_container:
+    image: $CONTAINER_REPO/windows_ci_mingw64:latest
+    cpu: $CPUS
+    memory: 4G
+  env:
+    CCACHE_DIR: C:/msys64/ccache
+    BUILD_DIR: "%CIRRUS_WORKING_DIR%/build"
+
+  ccache_cache:
+    folder: ${CCACHE_DIR}
+
+  mingw_info_script:
+    - C:\msys64\usr\bin\dash.exe -lc "where gcc"
+    - C:\msys64\usr\bin\dash.exe -lc "gcc --version"
+    - C:\msys64\usr\bin\dash.exe -lc "where perl"
+    - C:\msys64\usr\bin\dash.exe -lc "perl --version"
+
+  configure_script:
+    - C:\msys64\usr\bin\dash.exe -lc "mkdir %BUILD_DIR% &&
+      cd %BUILD_DIR% &&
+      %CIRRUS_WORKING_DIR%/configure
+        --enable-cassert
+        --enable-tap-tests
+        --with-icu
+        --with-libxml
+        --with-libxslt
+        --with-lz4
+        --enable-debug
+        CC='ccache gcc'
+        CXX='ccache g++'
+        CFLAGS='-Og -ggdb -pipe'
+        CXXFLAGS='-Og -ggdb'"
+
+  build_script:
+    C:\msys64\usr\bin\dash.exe -lc "cd %BUILD_DIR% && make -s world-bin -j${CPUS}"
+
+  upload_caches: ccache
+
+  tests_script:
+  - set "NoDefaultCurrentDirectoryInExePath=0"
+  - C:\msys64\usr\bin\dash.exe -lc "cd %BUILD_DIR% && make -s ${CHECK} ${CHECKFLAGS} -j${CPUS} TMPDIR=%BUILD_DIR%/tmp_install"
+
+  on_failure: *on_failure
 
 task:
   name: CompilerWarnings
-- 
2.17.1


--Ca0e2zgpnh8/XhnM
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment; filename="0002-f.patch"

From c8122c49502e9f95df392c41284f5e02819cf2cb Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Thu, 28 Jul 2022 10:57:26 -0500
Subject: [PATCH 2/2] f!

ci-os-only: mingw
---
 .cirrus.yml           | 23 ++++++++++++++++-------
 config/prep_buildtree | 18 ++++++------------
 2 files changed, 22 insertions(+), 19 deletions(-)

diff --git a/.cirrus.yml b/.cirrus.yml
index 1a06cdcaadb..93774485bef 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -348,8 +348,6 @@ WINDOWS_ENVIRONMENT_BASE: &WINDOWS_ENVIRONMENT_BASE
       # Avoids port conflicts between concurrent tap test runs
       PG_TEST_USE_UNIX_SOCKETS: 1
 
-    only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*windows.*'
-
     sysinfo_script: |
       chcp
       systeminfo
@@ -360,6 +358,8 @@ task:
   << : *WINDOWS_ENVIRONMENT_BASE
   name: Windows - Server 2019, VS 2019
 
+  only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*windows.*'
+
   env:
     # Our windows infrastructure doesn't have test concurrency above the level
     # of a single vcregress test target. Due to that, it's useful to run prove
@@ -465,6 +465,8 @@ task:
 task:
   << : *WINDOWS_ENVIRONMENT_BASE
   name: Windows - Server 2019, MinGW64
+  only_if: $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*mingw.*'
+
   windows_container:
     image: $CONTAINER_REPO/windows_ci_mingw64:latest
     cpu: $CPUS
@@ -482,21 +484,28 @@ task:
     - C:\msys64\usr\bin\dash.exe -lc "where perl"
     - C:\msys64\usr\bin\dash.exe -lc "perl --version"
 
+  setup_additional_packages_script: |
+    REM C:\msys64\usr\bin\pacman.exe -S --noconfirm busybox
+
   configure_script:
     - C:\msys64\usr\bin\dash.exe -lc "mkdir %BUILD_DIR% &&
-      cd %BUILD_DIR% &&
+      cd %BUILD_DIR% && for i in 1 2; do
       %CIRRUS_WORKING_DIR%/configure
+        --cache-file=${CCACHE_DIR}/configure.cache
         --enable-cassert
+        --enable-debug
         --enable-tap-tests
         --with-icu
         --with-libxml
         --with-libxslt
         --with-lz4
-        --enable-debug
         CC='ccache gcc'
         CXX='ccache g++'
         CFLAGS='-Og -ggdb -pipe'
-        CXXFLAGS='-Og -ggdb'"
+        CXXFLAGS='-Og -ggdb' && break;
+        rm -v ${CCACHE_DIR}/configure.cache;
+        done
+        "
 
   build_script:
     C:\msys64\usr\bin\dash.exe -lc "cd %BUILD_DIR% && make -s world-bin -j${CPUS}"
@@ -504,8 +513,8 @@ task:
   upload_caches: ccache
 
   tests_script:
-  - set "NoDefaultCurrentDirectoryInExePath=0"
-  - C:\msys64\usr\bin\dash.exe -lc "cd %BUILD_DIR% && make -s ${CHECK} ${CHECKFLAGS} -j${CPUS} TMPDIR=%BUILD_DIR%/tmp_install"
+    - set "NoDefaultCurrentDirectoryInExePath=0"
+    - C:\msys64\usr\bin\dash.exe -lc "cd %BUILD_DIR% && make -s ${CHECK} ${CHECKFLAGS} -j${CPUS} TMPDIR=%BUILD_DIR%/tmp_install"
 
   on_failure: *on_failure
 
diff --git a/config/prep_buildtree b/config/prep_buildtree
index a0eabd3dee2..4de30ac28a5 100644
--- a/config/prep_buildtree
+++ b/config/prep_buildtree
@@ -26,20 +26,14 @@ buildtree=`cd ${2:-'.'} && pwd`
 # If we did, it would interfere with installation of prebuilt docs from
 # the source tree, if a VPATH build is done from a distribution tarball.
 # See bug #5595.
-for item in `find "$sourcetree" -type d \( \( -name CVS -prune \) -o \( -name .git -prune \) -o -print \) | grep -v "$sourcetree/doc/src/sgml/\+"`; do
-    subdir=`expr "$item" : "$sourcetree\(.*\)"`
-    if test ! -d "$buildtree/$subdir"; then
-        mkdir -p "$buildtree/$subdir" || exit 1
-    fi
-done
+( cd "$sourcetree" && find . -type d \( \( -name CVS -prune \) -o \( -name .git -prune \) -o -print \) |grep -v "doc/src/sgml/\+" ) |
+	( cd "$buildtree" && xargs mkdir -p )
 
 for item in `find "$sourcetree" -name Makefile -print -o -name GNUmakefile -print | grep -v "$sourcetree/doc/src/sgml/images/"`; do
-    filename=`expr "$item" : "$sourcetree\(.*\)"`
-    if test ! -f "${item}.in"; then
-        if cmp "$item" "$buildtree/$filename" >/dev/null 2>&1; then : ; else
-            ln -fs "$item" "$buildtree/$filename" || exit 1
-        fi
-    fi
+    filename=${item#$sourcetree}
+    [ -e "$buildtree/$filename" ] && continue
+    cmp "$item" "$buildtree/$filename" >/dev/null 2>&1 ||
+        ln -fs "$item" "$buildtree/$filename"
 done
 
 exit 0
-- 
2.17.1


--Ca0e2zgpnh8/XhnM--