Subject: Re: CI and test improvements

From: Justin Pryzby <[email-masked]>
Date: Wed, 12 Jun 2024 08:10:23 -0500

On Fri, Jun 24, 2022 at 08:38:50AM +1200, Thomas Munro wrote:
> > I've also sent some patches to Thomas for cfbot to help progress some of these
> > patches (code coverage and documentation upload as artifacts).
> > https://github.com/justinpryzby/cfbot/commits/master
> 
> Thanks, sorry for lack of action, will get to these soon.

On Tue, Feb 13, 2024 at 01:10:21PM -0600, Justin Pryzby wrote:
> I sent various patches to cfbot but haven't heard back.

> https://www.postgresql.org/message-id/flat/[email-masked]
> https://www.postgresql.org/message-id/flat/[email-masked]
> https://github.com/justinpryzby/cfbot/commits/master
> https://github.com/macdice/cfbot/pulls

@Thomas: ping

I reintroduced the patch for ccache/windows -- v4.10 supports PCH, which
can make the builds 2x faster.
>From 8cebe912127ee5c5d03b62b231811c2670698da0 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <[email-masked]>
Date: Wed, 25 May 2022 21:53:22 -0500
Subject: [PATCH 1/6] cirrus/windows: add compiler_warnings_script

The goal is to fail due to warnings only after running tests.
(At least historically, it's too slow to run a separate windows VM to
compile with -Werror.)

https://www.postgresql.org/message-id/20220212212310.f645c6vw3njkgxka%40alap3.anarazel.de

I'm not sure how to write this test in windows shell; it's also easy to
write something that doesn't work in posix sh, since windows shell is
interpretting && and ||...

See also:
8a1ce5e54f6d144e4f8e19af7c767b026ee0c956
https://cirrus-ci.com/task/6241060062494720
https://cirrus-ci.com/task/6496366607204352

[email-masked]
[email-masked]
[email-masked]
[email-masked]
[email-masked]

ci-os-only: windows
---
 .cirrus.tasks.yml | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)

diff --git a/.cirrus.tasks.yml b/.cirrus.tasks.yml
index 33646faeadf..5a2b64f64c2 100644
--- a/.cirrus.tasks.yml
+++ b/.cirrus.tasks.yml
@@ -566,12 +566,21 @@ task:
 
   build_script: |
     vcvarsall x64
-    ninja -C build
+    ninja -C build |tee build.txt
+    REM Since pipes lose the exit status of the preceding command, rerun the compilation
+    REM without the pipe, exiting now if it fails, to avoid trying to run checks
+    ninja -C build > nul
 
   check_world_script: |
     vcvarsall x64
     meson test %MTEST_ARGS% --num-processes %TEST_JOBS%
 
+  # This should be last, so check_world is run even if there are warnings
+  always:
+    compiler_warnings_script:
+      # this avoids using metachars which would be interpretted by the windows shell
+      - sh -c 'if grep ": warning " build.txt; then exit 1; fi; exit 0'
+
   on_failure:
     <<: *on_failure_meson
     crashlog_artifacts:
-- 
2.42.0

>From 7f8e1c244b24f04a28d5a1a03da85f00419717e1 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <[email-masked]>
Date: Wed, 25 May 2022 22:05:13 -0500
Subject: [PATCH 2/6] cirrus/windows: ccache

https://www.postgresql.org/message-id/flat/20220522232606.GZ19626%40telsasoft.com

https://cirrus-ci.com/task/5213936495099904 build 29sec

, linux
linux-meson
ci-os-only: windows, windows-msvc
---
 .cirrus.tasks.yml | 21 ++++++++++++++++++---
 1 file changed, 18 insertions(+), 3 deletions(-)

diff --git a/.cirrus.tasks.yml b/.cirrus.tasks.yml
index 5a2b64f64c2..7a3fcc36fae 100644
--- a/.cirrus.tasks.yml
+++ b/.cirrus.tasks.yml
@@ -536,6 +536,11 @@ task:
   env:
     TEST_JOBS: 8 # wild guess, data based value welcome
 
+    CCACHE_DIR: $CIRRUS_WORKING_DIR/.ccache
+    CCACHE_MAXSIZE: "500M"
+    CCACHE_SLOPPINESS: pch_defines,time_macros
+    CCACHE_DEPEND: 1
+
     # Cirrus defaults to SetErrorMode(SEM_NOGPFAULTERRORBOX | ...). That
     # prevents crash reporting from working unless binaries do SetErrorMode()
     # themselves. Furthermore, it appears that either python or, more likely,
@@ -550,8 +555,11 @@ task:
   depends_on: SanityCheck
   only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*windows.*'
 
-  setup_additional_packages_script: |
-    REM choco install -y --no-progress ...
+  setup_additional_packages_script:
+    - choco install -y --no-progress ccache --version 4.10.0
+
+  ccache_cache:
+    folder: $CCACHE_DIR
 
   setup_hosts_file_script: |
     echo 127.0.0.1 pg-loadbalancetest >> c:\Windows\System32\Drivers\etc\hosts
@@ -562,7 +570,8 @@ task:
   # Use /DEBUG:FASTLINK to avoid high memory usage during linking
   configure_script: |
     vcvarsall x64
-    meson setup --backend ninja --buildtype debug -Dc_link_args=/DEBUG:FASTLINK -Dcassert=true -Dinjection_points=true -Db_pch=true -Dextra_lib_dirs=c:\openssl\1.1\lib -Dextra_include_dirs=c:\openssl\1.1\include -DTAR=%TAR% -DPG_TEST_EXTRA="%PG_TEST_EXTRA%" build
+    set CC=c:\ProgramData\chocolatey\lib\ccache\tools\ccache-4.10-windows-x86_64\ccache.exe cl.exe
+    meson setup build --backend ninja --buildtype debug -Dc_link_args=/DEBUG:FASTLINK -Dcassert=true -Dinjection_points=true -Db_pch=true -Dextra_lib_dirs=c:\openssl\1.1\lib -Dextra_include_dirs=c:\openssl\1.1\include -DTAR=%TAR% -DPG_TEST_EXTRA="%PG_TEST_EXTRA%" -Dc_args="/Z7"
 
   build_script: |
     vcvarsall x64
@@ -571,6 +580,12 @@ task:
     REM without the pipe, exiting now if it fails, to avoid trying to run checks
     ninja -C build > nul
 
+    ccache --show-log-stats -v
+    ccache --show-stats -v
+
+  always:
+    upload_caches: [ccache]
+
   check_world_script: |
     vcvarsall x64
     meson test %MTEST_ARGS% --num-processes %TEST_JOBS%
-- 
2.42.0

>From 7ad1d5c1999210fc2a4298d4afc756dac78eb4a6 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <[email-masked]>
Date: Fri, 25 Nov 2022 13:57:17 -0600
Subject: [PATCH 3/6] WIP: ci/meson: allow showing only failed tests ..

It's simpler and seems to make more sense to integrate this with
testwrap, rather than to run it after check-world, but only if it
failed, and finding a way to preserve the exit code.

https://www.postgresql.org/message-id/[email-masked]

> It is wasteful to upload thousdands of logfiles to show a single
> failure.  That would make our cirrus tasks faster - compressing and
> uploading the logs takes over a minute.
>
> It's also a lot friendlier to show fewer than 8 pages of test folders to
> search through to find the one that failed.

macos
linux-meson
ci-os-only: freebsd
---
 .cirrus.tasks.yml  | 14 ++++++++------
 src/tools/testwrap |  6 ++++++
 2 files changed, 14 insertions(+), 6 deletions(-)

diff --git a/.cirrus.tasks.yml b/.cirrus.tasks.yml
index 7a3fcc36fae..e9f3a4fb8f9 100644
--- a/.cirrus.tasks.yml
+++ b/.cirrus.tasks.yml
@@ -21,6 +21,7 @@ env:
   PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests
   TEMP_CONFIG: ${CIRRUS_WORKING_DIR}/src/tools/ci/pg_ci_base.conf
   PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance
+  PG_FAILED_TESTDIR: ${CIRRUS_WORKING_DIR}/failed.build

 # What files to preserve in case tests fail
@@ -35,9 +36,9 @@ on_failure_ac: &on_failure_ac
 on_failure_meson: &on_failure_meson
   testrun_artifacts:
     paths:
-      - "build*/testrun/**/*.log"
-      - "build*/testrun/**/*.diffs"
-      - "build*/testrun/**/regress_log_*"
+      - failed.build*/**/*.log
+      - failed.build*/**/*.diffs
+      - failed.build*/**/regress_log_*
     type: text/plain
 
   # In theory it'd be nice to upload the junit files meson generates, so that
@@ -136,6 +137,7 @@ task:
     CPPFLAGS: -DRELCACHE_FORCE_RELEASE -DCOPY_PARSE_PLAN_TREES -DWRITE_READ_PARSE_PLAN_TREES -DRAW_EXPRESSION_COVERAGE_TEST -DENFORCE_REGRESSION_TEST_NAME_RESTRICTIONS
     CFLAGS: -Og -ggdb
 
+    PG_FAILED_TESTDIR: ${CIRRUS_WORKING_DIR}/failed.build
     PG_TEST_PG_UPGRADE_MODE: --link
 
   <<: *freebsd_task_template
@@ -194,10 +196,10 @@ task:
       ulimit -c unlimited
       meson test $MTEST_ARGS --quiet --suite setup
       export LD_LIBRARY_PATH="$(pwd)/build/tmp_install/usr/local/pgsql/lib/:$LD_LIBRARY_PATH"
-      mkdir -p build/testrun
+      mkdir -p build/testrun ${PG_FAILED_TESTDIR}
       build/tmp_install/usr/local/pgsql/bin/initdb -N build/runningcheck --no-instructions -A trust
       echo "include '$(pwd)/src/tools/ci/pg_ci_base.conf'" >> build/runningcheck/postgresql.conf
-      build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start
+      build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l ${PG_FAILED_TESTDIR}/runningcheck.log start
       meson test $MTEST_ARGS --num-processes ${TEST_JOBS} --setup running
       build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop
     EOF
@@ -400,7 +402,7 @@ task:
       test_world_32_script: |
         su postgres <<-EOF
           ulimit -c unlimited
-          PYTHONCOERCECLOCALE=0 LANG=C meson test $MTEST_ARGS -C build-32 --num-processes ${TEST_JOBS}
+          PYTHONCOERCECLOCALE=0 LANG=C PG_FAILED_TESTDIR=`pwd`/failed.build-32 meson test $MTEST_ARGS -C build-32 --num-processes ${TEST_JOBS}
         EOF
 
       on_failure:
diff --git a/src/tools/testwrap b/src/tools/testwrap
index d01e61051cb..7e5a17dbf94 100755
--- a/src/tools/testwrap
+++ b/src/tools/testwrap
@@ -49,4 +49,10 @@ if sp.returncode == 0:
 else:
     print('# test failed')
     open(os.path.join(testdir, 'test.fail'), 'x')
+    faileddir = os.getenv('PG_FAILED_TESTDIR')
+    if faileddir:
+        parentdir = os.path.dirname(testdir)
+        newdest = os.path.join(faileddir, os.path.basename(parentdir), os.path.basename(testdir))
+        shutil.copytree(testdir, newdest)
+
 sys.exit(sp.returncode)
-- 
2.42.0

>From dcc133e30bca653e31f8672c6e822a55b9c0e550 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <[email-masked]>
Date: Sat, 26 Feb 2022 19:39:10 -0600
Subject: [PATCH 4/6] WIP: cirrus: upload changed html docs as artifacts

This could be done on the client side (cfbot).  One advantage of doing
it here is that fewer docs are uploaded - many patches won't upload docs
at all.

https://www.postgresql.org/message-id/flat/[email-masked]
https://www.postgresql.org/message-id/CAB8KJ=[email-masked]

https://cirrus-ci.com/task/5396696388599808

//-os-only:
---
 .cirrus.tasks.yml              | 25 ++++++++++++++++++++++++-
 src/tools/ci/copy-changed-docs | 29 +++++++++++++++++++++++++++++
 2 files changed, 53 insertions(+), 1 deletion(-)
 create mode 100755 src/tools/ci/copy-changed-docs

diff --git a/.cirrus.tasks.yml b/.cirrus.tasks.yml
index e9f3a4fb8f9..77dca9e3483 100644
--- a/.cirrus.tasks.yml
+++ b/.cirrus.tasks.yml
@@ -23,6 +23,15 @@ env:
   PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance
   PG_FAILED_TESTDIR: ${CIRRUS_WORKING_DIR}/failed.build
 
+  # The commit that this branch is rebased on.  There's no easy way to find this.
+  # This does the right thing for cfbot, which always squishes all patches into a single commit.
+  # And does the right thing for any 1-patch commits.
+  # Patch series manually submitted to cirrus would benefit from setting this to the
+  # number of patches in the series (or directly to the commit the series was rebased on).
+  #BASE_COMMIT: HEAD~1
+  # For demo purposes:
+  BASE_COMMIT: HEAD~55
+
 
 # What files to preserve in case tests fail
 on_failure_ac: &on_failure_ac
@@ -772,7 +781,7 @@ task:
       time make -s -j${BUILD_JOBS} world-bin
 
   ###
-  # Verify docs can be built
+  # Verify docs can be built, and upload changed docs as artifacts
   ###
   # XXX: Only do this if there have been changes in doc/ since last build
   always:
@@ -784,6 +793,20 @@ task:
         CLANG="ccache clang"
       make -s -j${BUILD_JOBS} clean
       time make -s -j${BUILD_JOBS} -C doc
+      cp -r doc new-docs
+
+      # Re-build HTML docs from the base commit.
+      git checkout "$BASE_COMMIT" -- doc
+      make -s -C doc clean
+      time make -s -C doc html
+      cp -r doc old-docs
+
+    copy_changed_docs_script:
+      - src/tools/ci/copy-changed-docs "old-docs" "new-docs" "html_docs"
+
+    html_docs_artifacts:
+      paths: ['html_docs/*.html', 'html_docs/*.png', 'html_docs/*.css']
+
 
   ###
   # Verify headerscheck / cpluspluscheck succeed
diff --git a/src/tools/ci/copy-changed-docs b/src/tools/ci/copy-changed-docs
new file mode 100755
index 00000000000..1c921a8df6f
--- /dev/null
+++ b/src/tools/ci/copy-changed-docs
@@ -0,0 +1,29 @@
+#! /bin/sh
+# Copy HTML which differ between $old and $new into $outdir
+set -e
+
+old=$1
+new=$2
+outdir=$3
+
+# The index is large and changes often
+skippages="bookindex.html"
+
+mkdir "$outdir"
+cp "$new"/src/sgml/html/*.css "$new"/src/sgml/html/*.svg "$outdir"
+
+changed=`git diff --no-index --name-only "$old"/src/sgml/html "$new"/src/sgml/html` ||
+	[ $? -eq 1 ]
+
+for f in $changed
+do
+	# Avoid removed files
+	[ -f "$f" ] || continue
+
+	echo "$f" |grep -Ew "$skippages" >/dev/null &&
+		continue
+
+	cp -v "$f" "$outdir"
+done
+
+exit 0
-- 
2.42.0

>From 5cedbf6bbfecf0cd82f6116850528c2cd75f4b0a Mon Sep 17 00:00:00 2001
From: Justin Pryzby <[email-masked]>
Date: Mon, 28 Feb 2022 23:18:19 -0600
Subject: [PATCH 5/6] +html index file

This allows linking to the artifacts from the last successful build.

//freebsd
ci-os-only: warnings
---
 src/tools/ci/copy-changed-docs | 39 ++++++++++++++++++++++++++++++++--
 1 file changed, 37 insertions(+), 2 deletions(-)

diff --git a/src/tools/ci/copy-changed-docs b/src/tools/ci/copy-changed-docs
index 1c921a8df6f..0efad386cca 100755
--- a/src/tools/ci/copy-changed-docs
+++ b/src/tools/ci/copy-changed-docs
@@ -5,6 +5,7 @@ set -e
 old=$1
 new=$2
 outdir=$3
+branch=$CIRRUS_BRANCH
 
 # The index is large and changes often
 skippages="bookindex.html"
@@ -12,6 +13,19 @@ skippages="bookindex.html"
 mkdir "$outdir"
 cp "$new"/src/sgml/html/*.css "$new"/src/sgml/html/*.svg "$outdir"
 
+# The index is useful to allow a static link (not specific to a cirrus run) to the artifacts for the most-recent, successful CI run for a branch
+branchurl=https://api.cirrus-ci.com/v1/artifact/github/$CIRRUS_REPO_FULL_NAME/$CIRRUS_TASK_NAME/html_docs/html_docs/00-index.html?branch=$branch
+
+index="$outdir/00-index.html"
+cat >"$index" <<EOF
+<html>
+<head><title>Index of docs changed since: $BASE_COMMIT</title></head>
+<body>
+<!-- A link to documentation for the most recent successful build: $branchurl -->
+<h1>Index of docs changed since: $BASE_COMMIT</h1>
+<ul>
+EOF
+
 changed=`git diff --no-index --name-only "$old"/src/sgml/html "$new"/src/sgml/html` ||
 	[ $? -eq 1 ]
 
@@ -23,7 +37,28 @@ do
 	echo "$f" |grep -Ew "$skippages" >/dev/null &&
 		continue
 
-	cp -v "$f" "$outdir"
-done
+	cp -v "$f" "$outdir" >&2
+	fn=${f##*/}
+	# ?branch=... is needed when accessing the artifacts for the static link for the branch
+	# It's not needed and ignored if accessing artifacts for *this* CI run
+	echo "<li><a href='$fn?branch=$branch'>$fn</a>"
+done >>"$index"
+
+github=https://github.com/$CIRRUS_REPO_FULL_NAME/commit
+cirrus=https://cirrus-ci.com/build
+
+cat >>"$index" <<EOF
+</ul>
+<hr>
+<code>
+<br>This file was written on: `date --rfc-822 --utc`
+<br>CIRRUS_CHANGE_TITLE: $CIRRUS_CHANGE_TITLE
+<br>CIRRUS_CHANGE_IN_REPO: <a href="$github/$CIRRUS_CHANGE_IN_REPO">$CIRRUS_CHANGE_IN_REPO</a>
+<br>CIRRUS_BUILD_ID: <a href="$cirrus/$CIRRUS_BUILD_ID">$CIRRUS_BUILD_ID</a>
+<br>CIRRUS_LAST_GREEN_CHANGE: <a href="$github/$CIRRUS_LAST_GREEN_CHANGE">$CIRRUS_LAST_GREEN_CHANGE</a>
+<br>CIRRUS_LAST_GREEN_BUILD_ID: <a href="$cirrus/$CIRRUS_LAST_GREEN_BUILD_ID">$CIRRUS_LAST_GREEN_BUILD_ID</a>
+</code>
+</body></html>
+EOF
 
 exit 0
-- 
2.42.0

>From b084963b13e34a977c54a9b8f9e02543006720c1 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <[email-masked]>
Date: Wed, 19 Oct 2022 20:33:28 -0500
Subject: [PATCH 6/6] WIP: show changed docs with meson

//-os-only:
---
 .cirrus.tasks.yml              | 26 ++++++++++++++++++++++----
 src/tools/ci/copy-changed-docs |  3 ++-
 2 files changed, 24 insertions(+), 5 deletions(-)

diff --git a/.cirrus.tasks.yml b/.cirrus.tasks.yml
index 77dca9e3483..f4a176103e3 100644
--- a/.cirrus.tasks.yml
+++ b/.cirrus.tasks.yml
@@ -786,20 +786,38 @@ task:
   # XXX: Only do this if there have been changes in doc/ since last build
   always:
     docs_build_script: |
-      time ./configure \
+      mkdir build-autoconf
+      cd build-autoconf
+      time ../configure \
         --cache gcc.cache \
+        --without-icu \
         CC="ccache gcc" \
         CXX="ccache g++" \
         CLANG="ccache clang"
       make -s -j${BUILD_JOBS} clean
       time make -s -j${BUILD_JOBS} -C doc
-      cp -r doc new-docs
+      cp -r doc ../new-docs
 
       # Re-build HTML docs from the base commit.
-      git checkout "$BASE_COMMIT" -- doc
+      git checkout "$BASE_COMMIT" -- ../doc
       make -s -C doc clean
       time make -s -C doc html
-      cp -r doc old-docs
+      cp -r doc ../old-docs
+
+    # Exercise HTML and other docs:
+    ninja_docs_build_script: |
+      make maintainer-clean # XXX not needed once compiler-warnings is switched meson
+      mkdir build-ninja
+      cd build-ninja
+      time meson setup
+      time ninja docs
+      cp -r doc ../new-docs
+
+      # Re-build HTML docs from the base commit.
+      git checkout "$BASE_COMMIT" -- ../doc
+      ninja clean
+      time ninja doc/src/sgml/html
+      cp -r doc ../old-docs
 
     copy_changed_docs_script:
       - src/tools/ci/copy-changed-docs "old-docs" "new-docs" "html_docs"
diff --git a/src/tools/ci/copy-changed-docs b/src/tools/ci/copy-changed-docs
index 0efad386cca..ee3135c298b 100755
--- a/src/tools/ci/copy-changed-docs
+++ b/src/tools/ci/copy-changed-docs
@@ -11,7 +11,8 @@ branch=$CIRRUS_BRANCH
 skippages="bookindex.html"
 
 mkdir "$outdir"
-cp "$new"/src/sgml/html/*.css "$new"/src/sgml/html/*.svg "$outdir"
+cp doc/src/sgml/*.css "$outdir"
+cp doc/src/sgml/images/*.svg "$outdir"
 
 # The index is useful to allow a static link (not specific to a cirrus run) to the artifacts for the most-recent, successful CI run for a branch
 branchurl=https://api.cirrus-ci.com/v1/artifact/github/$CIRRUS_REPO_FULL_NAME/$CIRRUS_TASK_NAME/html_docs/html_docs/00-index.html?branch=$branch
-- 
2.42.0

---

From: Nazir Bilal Yavuz <[email-masked]>
Date: Thu, 13 Jun 2024 14:38:46 +0300

Hi,

Thanks for working on this!

> can make the builds 2x faster.

I applied 0001 and 0002 to see ccache support on Windows but the build
step failed with: 'ccache: error: No stats log has been configured'.
Perhaps you forgot to add 'CCACHE_STATSLOG: $CCACHE_DIR.stats.log' to
0002? After adding that line, CI finished successfully. And, I confirm
that the build step takes ~30 seconds now; it was ~90 seconds before
that.

-- 
Regards,
Nazir Bilal Yavuz
Microsoft

---

From: Justin Pryzby <[email-masked]>
Date: Thu, 13 Jun 2024 06:56:20 -0500

On Thu, Jun 13, 2024 at 02:38:46PM +0300, Nazir Bilal Yavuz wrote:
> 0002?

Something like that - I put the line back.  I don't know if statslog
should be included in the patch, but it's useful for demonstrating that
it's working.

ccache should be installed in the image rather than re-installed on each
invocation.

-- 
Justin
>From ad03da2cfa3ecf23334f8b31f2e4529ba3094512 Mon Sep 17 00:00:00 2001

See also:

ci-os-only: windows
---

   build_script: |
     vcvarsall x64
-    ninja -C build
 
   check_world_script: |
     vcvarsall x64

>From 31629f2f413ceb5f2704ac71e24d0d8e82cc26ab Mon Sep 17 00:00:00 2001
From: Justin Pryzby <[email-masked]>
Date: Wed, 25 May 2022 22:05:13 -0500
Subject: [PATCH 2/6] WIP: cirrus/windows: ccache

ccache v4.10 supports PCH, so this now seems viable.
(ccache 4.7 added support for depend mode with MSVC).

ccache should be installed in the VM rather than with choco in each
invocation.

, linux
linux-meson
ci-os-only: windows, windows-msvc
---
 .cirrus.tasks.yml | 22 +++++++++++++++++++---
 1 file changed, 19 insertions(+), 3 deletions(-)

diff --git a/.cirrus.tasks.yml b/.cirrus.tasks.yml
index 5a2b64f64c2..d8c2a396a20 100644
--- a/.cirrus.tasks.yml
+++ b/.cirrus.tasks.yml
@@ -536,6 +536,12 @@ task:
   env:
     TEST_JOBS: 8 # wild guess, data based value welcome
 
+    CCACHE_DIR: $CIRRUS_WORKING_DIR/.ccache
+    CCACHE_MAXSIZE: "500M"
+    CCACHE_SLOPPINESS: pch_defines,time_macros
+    CCACHE_DEPEND: 1
+    CCACHE_STATSLOG: $CCACHE_DIR.stats.log
+
     # Cirrus defaults to SetErrorMode(SEM_NOGPFAULTERRORBOX | ...). That
     # prevents crash reporting from working unless binaries do SetErrorMode()
     # themselves. Furthermore, it appears that either python or, more likely,
@@ -550,8 +556,11 @@ task:
   depends_on: SanityCheck
   only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*windows.*'

   setup_hosts_file_script: |
     echo 127.0.0.1 pg-loadbalancetest >> c:\Windows\System32\Drivers\etc\hosts
@@ -562,7 +571,8 @@ task:
   # Use /DEBUG:FASTLINK to avoid high memory usage during linking
   configure_script: |
     vcvarsall x64
-    meson setup --backend ninja --buildtype debug -Dc_link_args=/DEBUG:FASTLINK -Dcassert=true -Dinjection_points=true -Db_pch=true -Dextra_lib_dirs=c:\openssl\1.1\lib -Dextra_include_dirs=c:\openssl\1.1\include -DTAR=%TAR% -DPG_TEST_EXTRA="%PG_TEST_EXTRA%" build
+    set CC=c:\ProgramData\chocolatey\lib\ccache\tools\ccache-4.10-windows-x86_64\ccache.exe cl.exe
+    meson setup build --backend ninja --buildtype debugoptimized -Dc_link_args=/DEBUG:FASTLINK -Dcassert=true -Dinjection_points=true -Db_pch=true -Dextra_lib_dirs=c:\openssl\1.1\lib -Dextra_include_dirs=c:\openssl\1.1\include -DTAR=%TAR% -DPG_TEST_EXTRA="%PG_TEST_EXTRA%" -Dc_args="/Z7"
 
   build_script: |
     vcvarsall x64
@@ -571,6 +581,12 @@ task:
     REM without the pipe, exiting now if it fails, to avoid trying to run checks
     ninja -C build > nul

>From 24280295d5c4c5c62242de1db1ab8e7582097899 Mon Sep 17 00:00:00 2001

macos
linux-meson
ci-os-only: freebsd
---

index d8c2a396a20..3df51e6c896 100644

 

       on_failure:

>From b164129479b56d2b56ee8bacecb3ad31e5c4a914 Mon Sep 17 00:00:00 2001

//-os-only:
---

diff --git a/.cirrus.tasks.yml b/.cirrus.tasks.yml
index 3df51e6c896..3f1960c4cf9 100644
--- a/.cirrus.tasks.yml
+++ b/.cirrus.tasks.yml
@@ -23,6 +23,15 @@ env:
   PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance
   PG_FAILED_TESTDIR: ${CIRRUS_WORKING_DIR}/failed.build

 # What files to preserve in case tests fail
 on_failure_ac: &on_failure_ac
@@ -773,7 +782,7 @@ task:
       time make -s -j${BUILD_JOBS} world-bin
 
   ###
@@ -785,6 +794,20 @@ task:
 
   ###

>From 2fa279d3596976b0face797d16e4f28a357bb73e Mon Sep 17 00:00:00 2001

//freebsd
ci-os-only: warnings
---

 
 
@@ -23,7 +37,28 @@ do
 
-	cp -v "$f" "$outdir"
-done
 
 exit 0
-- 
2.42.0

>From a3e1a122e90cfc6b76623240c45a81e67a760670 Mon Sep 17 00:00:00 2001

//-os-only:
---

diff --git a/.cirrus.tasks.yml b/.cirrus.tasks.yml
index 3f1960c4cf9..8fa1bf54985 100644
--- a/.cirrus.tasks.yml
+++ b/.cirrus.tasks.yml
@@ -787,20 +787,38 @@ task:
   # XXX: Only do this if there have been changes in doc/ since last build
   always:
     docs_build_script: |
-      time ./configure \
+      mkdir build-autoconf
+      cd build-autoconf
+      time ../configure \
         --cache gcc.cache \
+        --without-icu \
         CC="ccache gcc" \
         CXX="ccache g++" \
         CLANG="ccache clang"
       make -s -j${BUILD_JOBS} clean
       time make -s -j${BUILD_JOBS} -C doc
-      cp -r doc new-docs
+      cp -r doc ../new-docs

 
 mkdir "$outdir"

---

From: Michael Paquier <[email-masked]>
Date: Fri, 14 Jun 2024 08:05:09 +0900

> ccache should be installed in the image rather than re-installed on each

Getting a 90s -> 30s improvement is nice.  With such numbers, 0002 is
worth considering first.

In 0001, how OK is it to rely on the existence of tee for the VS2019
environments?  The base images include it, meaning that it is OK?

I'd rather keep this line in 0002, as a matter of documentation.

As of https://docs.mesa3d.org/meson.html#compiler-specification, using
CC is supported by meson (didn't know that), but shouldn't this be set
in the "env:" part of the VS2019 task in .cirrus.tasks.yml?
--
Michael

---

From: Nazir Bilal Yavuz <[email-masked]>
Date: Fri, 14 Jun 2024 17:36:54 +0300

Hi,

> ccache should be installed in the image rather than re-installed on each

ccache is installed in the Windows VM images now [1]. It can be used
as 'set CC=ccache.exe cl.exe' in the Windows CI task.

[1] https://github.com/anarazel/pg-vm-images/commit/03a9225ac962fb30b5c0722c702941e2d7c1e81e

-- 
Regards,
Nazir Bilal Yavuz
Microsoft