From: Justin Pryzby <pryzby@telsasoft.com>
To: Andres Freund <andres@anarazel.de>
Cc: Andrew Dunstan <andrew@dunslane.net>
Cc: Thomas Munro <thomas.munro@gmail.com>
Cc: pgsql-hackers@lists.postgresql.org, Noah Misch <noah@leadboat.com>
Cc: Michael Paquier <michael@paquier.xyz>
Cc: Anastasia Lubennikova <lubennikovaav@gmail.com>
Cc: Tom Lane <tgl@sss.pgh.pa.us>
Cc: Robert Haas <robertmhaas@gmail.com>
Cc: Melanie Plageman <melanieplageman@gmail.com>
Cc: Peter Eisentraut <peter.eisentraut@enterprisedb.com>
Cc: Daniel Gustafsson <daniel@yesql.se>
Cc: samay sharma <smilingsamay@gmail.com>
Subject: Re: CI and test improvements
Date: Wed, 4 Jan 2023 17:44:24 -0600
Message-ID: <20230104234424.GE3109@telsasoft.com> (raw)
In-Reply-To: <20221122225744.GF11463@telsasoft.com>
References: <20220828144447.GA21897@telsasoft.com>
<20220828160752.l5l66k3eptokzhzj@awork3.anarazel.de>
<20220828171029.GO2342@telsasoft.com>
<20220828212802.r6eymfffrgr3lxxt@awork3.anarazel.de>
<20220910200542.GX31833@telsasoft.com>
<20221104235412.GE16921@telsasoft.com>
<20221105015946.yrxijqb7h4rqhp6d@awork3.anarazel.de>
<20221113235303.GA26337@telsasoft.com>
<20221121224542.p2zapvyvb7objluw@alap3.anarazel.de>
<20221122225744.GF11463@telsasoft.com>
On Tue, Nov 22, 2022 at 04:57:44PM -0600, Justin Pryzby wrote:
> I shuffled my branch around and sending now the current "docs" patches,
> but I suppose this is waiting on the "convert CompilerWarnings task to
> meson" patch.
In case it's not, here's a version to do that now.
Attachments:download all (7 files, .tar.gz)0001-cirrus-windows-add-compiler_warnings_script.patch (text/x-diff, 1.6K) download
From 59bbbce620fdf3c4f228c0569b1fac29beb06988 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Wed, 25 May 2022 21:53:22 -0500
Subject: [PATCH 1/7] cirrus/windows: add compiler_warnings_script
I'm not sure how to write this test in windows shell; it's also easy to
write something that doesn't work in posix sh, since windows shell is
interpretting && and ||...
https://www.postgresql.org/message-id/20220212212310.f645c6vw3njkgxka%40alap3.anarazel.de
See also:
8a1ce5e54f6d144e4f8e19af7c767b026ee0c956
https://cirrus-ci.com/task/6241060062494720
https://cirrus-ci.com/task/6496366607204352
ci-os-only: windows
---
.cirrus.yml | 11 ++++++++++-
1 file changed, 10 insertions(+), 1 deletion(-)
diff --git a/.cirrus.yml b/.cirrus.ymlindex 69837bcd5ad..fd461048372 100644--- a/.cirrus.yml+++ b/.cirrus.yml@@ -564,12 +564,21 @@ task:
build_script: |
vcvarsall x64
- ninja -C build+ ninja -C build |tee build.txt+ REM Since pipes lose the exit status of the preceding command, rerun the compilation+ REM without the pipe, exiting now if it fails, to avoid trying to run checks+ ninja -C build > nul
check_world_script: |
vcvarsall x64
meson test %MTEST_ARGS% --num-processes %TEST_JOBS%
+ # This should be last, so check_world is run even if there are warnings+ always:+ compiler_warnings_script:+ # this avoids using metachars which would be interpretted by the windows shell+ - sh -c 'if grep ": warning " build.txt; then exit 1; fi; exit 0'+
on_failure:
<<: *on_failure_meson
crashlog_artifacts:
--
2.25.1
0002-cirrus-macos-update-to-macos-ventura.patch (text/x-diff, 933B) download
From 4edbdf0337c75bd69e536b542a4cd06dac87c33d Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Fri, 4 Nov 2022 15:43:56 -0500
Subject: [PATCH 2/7] cirrus/macos: update to macos ventura
ci-os-only: macos
---
.cirrus.yml | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/.cirrus.yml b/.cirrus.ymlindex fd461048372..858454a3d08 100644--- a/.cirrus.yml+++ b/.cirrus.yml@@ -404,7 +404,7 @@ task:
task:
- name: macOS - Monterey - Meson+ name: macOS - Ventura - Meson
env:
CPUS: 4 # always get that much for cirrusci macOS instances
@@ -429,7 +429,7 @@ task:
only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*(macos|darwin|osx).*'
macos_instance:
- image: ghcr.io/cirruslabs/macos-monterey-base:latest+ image: ghcr.io/cirruslabs/macos-ventura-base:latest
sysinfo_script: |
id
--
2.25.1
0003-cirrus-freebsd-run-with-more-CPUs-RAM-and-do-not-rep.patch (text/x-diff, 1.8K) download
From 2f6a7102f717d7fe5e291d6e439fd32b122180a1 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Fri, 24 Jun 2022 00:09:12 -0500
Subject: [PATCH 3/7] cirrus/freebsd: run with more CPUs+RAM and do not
repartition
There was some historic problem where tests under freebsd took 8+ minutes (and
before 4a288a37f took 15 minutes).
This reduces test time from 10min to 3min.
4 CPUs 4 tests https://cirrus-ci.com/task/4880240739614720
4 CPUs 6 tests https://cirrus-ci.com/task/4664440120410112 https://cirrus-ci.com/task/4586784884523008
4 CPUs 8 tests https://cirrus-ci.com/task/5001995491737600
6 CPUs https://cirrus-ci.com/task/6678321684545536
8 CPUs https://cirrus-ci.com/task/6264854121021440
See also:
https://www.postgresql.org/message-id/flat/20220310033347.hgxk4pyarzq4hxwp@alap3.anarazel.de#f36c0b17e33e31e7925e7e5812998686
8 jobs 7min https://cirrus-ci.com/task/6186376667332608
//-os-only: freebsd
---
.cirrus.yml | 10 +++-------
1 file changed, 3 insertions(+), 7 deletions(-)
diff --git a/.cirrus.yml b/.cirrus.ymlindex 858454a3d08..2ba6b7cc2d8 100644--- a/.cirrus.yml+++ b/.cirrus.yml@@ -131,11 +131,9 @@ task:
name: FreeBSD - 13 - Meson
env:
- # FreeBSD on GCP is slow when running with larger number of CPUS /- # jobs. Using one more job than cpus seems to work best.- CPUS: 2- BUILD_JOBS: 3- TEST_JOBS: 3+ CPUS: 4+ BUILD_JOBS: 4+ TEST_JOBS: 6
CCACHE_DIR: /tmp/ccache_dir
CPPFLAGS: -DRELCACHE_FORCE_RELEASE -DCOPY_PARSE_PLAN_TREES -DWRITE_READ_PARSE_PLAN_TREES -DRAW_EXPRESSION_COVERAGE_TEST
@@ -160,8 +158,6 @@ task:
ccache_cache:
folder: $CCACHE_DIR
- # Work around performance issues due to 32KB block size- repartition_script: src/tools/ci/gcp_freebsd_repartition.sh
create_user_script: |
pw useradd postgres
chown -R postgres:postgres .
--
2.25.1
0004-cirrus-freebsd-define-ENFORCE_REGRESSION_TEST_NAME_R.patch (text/x-diff, 932B) download
From b55c042ec467a03984a5e2132dc4e7fe0f49fef3 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Fri, 9 Dec 2022 15:32:50 -0600
Subject: [PATCH 4/7] cirrus/freebsd: define
ENFORCE_REGRESSION_TEST_NAME_RESTRICTIONS
See also: 54100f5c6052404f68de9ce7310ceb61f1c291f8
ci-os-only: freebsd
---
.cirrus.yml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/.cirrus.yml b/.cirrus.ymlindex 2ba6b7cc2d8..34c9d52b624 100644--- a/.cirrus.yml+++ b/.cirrus.yml@@ -136,7 +136,7 @@ task:
TEST_JOBS: 6
CCACHE_DIR: /tmp/ccache_dir
- CPPFLAGS: -DRELCACHE_FORCE_RELEASE -DCOPY_PARSE_PLAN_TREES -DWRITE_READ_PARSE_PLAN_TREES -DRAW_EXPRESSION_COVERAGE_TEST+ CPPFLAGS: -DRELCACHE_FORCE_RELEASE -DCOPY_PARSE_PLAN_TREES -DWRITE_READ_PARSE_PLAN_TREES -DRAW_EXPRESSION_COVERAGE_TEST -DENFORCE_REGRESSION_TEST_NAME_RESTRICTIONS
CFLAGS: -Og -ggdb
depends_on: SanityCheck
--
2.25.1
0005-cirrus-warnings-use-a-single-common-always-block.patch (text/x-diff, 3.2K) download
From a4ecb1a1bf008052108382f5833345680eec156c Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Tue, 19 Jul 2022 12:38:45 -0500
Subject: [PATCH 5/7] cirrus/warnings: use a single/common 'always' block
ci-os-only: warnings
---
.cirrus.yml | 35 ++++++++++++++---------------------
1 file changed, 14 insertions(+), 21 deletions(-)
diff --git a/.cirrus.yml b/.cirrus.ymlindex 34c9d52b624..123f6957513 100644--- a/.cirrus.yml+++ b/.cirrus.yml@@ -696,8 +696,8 @@ task:
# different compilers to build with different combinations of dtrace on/off
# and cassert on/off.
- # gcc, cassert off, dtrace on
always:
+ # gcc, cassert off, dtrace on
gcc_warning_script: |
time ./configure \
--cache gcc.cache \
@@ -707,8 +707,7 @@ task:
make -s -j${BUILD_JOBS} clean
time make -s -j${BUILD_JOBS} world-bin
- # gcc, cassert on, dtrace off- always:+ # gcc, cassert on, dtrace off
gcc_a_warning_script: |
time ./configure \
--cache gcc.cache \
@@ -718,8 +717,7 @@ task:
make -s -j${BUILD_JOBS} clean
time make -s -j${BUILD_JOBS} world-bin
- # clang, cassert off, dtrace off- always:+ # clang, cassert off, dtrace off
clang_warning_script: |
time ./configure \
--cache clang.cache \
@@ -728,8 +726,7 @@ task:
make -s -j${BUILD_JOBS} clean
time make -s -j${BUILD_JOBS} world-bin
- # clang, cassert on, dtrace on- always:+ # clang, cassert on, dtrace on
clang_a_warning_script: |
time ./configure \
--cache clang.cache \
@@ -740,8 +737,7 @@ task:
make -s -j${BUILD_JOBS} clean
time make -s -j${BUILD_JOBS} world-bin
- # cross-compile to windows- always:+ # cross-compile to windows
mingw_cross_warning_script: |
time ./configure \
--host=x86_64-w64-mingw32 \
@@ -755,7 +751,6 @@ task:
# Verify docs can be built
###
# XXX: Only do this if there have been changes in doc/ since last build
- always:
docs_build_script: |
time ./configure \
--cache gcc.cache \
@@ -765,16 +760,15 @@ task:
make -s -j${BUILD_JOBS} clean
time make -s -j${BUILD_JOBS} -C doc
- ###- # Verify headerscheck / cpluspluscheck succeed- #- # - Don't use ccache, the files are uncacheable, polluting ccache's- # cache- # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose- # - XXX have to disable ICU to avoid errors:- # https://postgr.es/m/20220323002024.f2g6tivduzrktgfa%40alap3.anarazel.de- ###- always:+ ###+ # Verify headerscheck / cpluspluscheck succeed+ #+ # - Don't use ccache, the files are uncacheable, polluting ccache's+ # cache+ # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose+ # - XXX have to disable ICU to avoid errors:+ # https://postgr.es/m/20220323002024.f2g6tivduzrktgfa%40alap3.anarazel.de+ ###
headers_headerscheck_script: |
time ./configure \
${LINUX_CONFIGURE_FEATURES} \
@@ -786,5 +780,4 @@ task:
headers_cpluspluscheck_script: |
time make -s cpluspluscheck EXTRAFLAGS='-fmax-errors=10'
- always:
upload_caches: ccache
--
2.25.1
0006-cirrus-upload-changed-html-docs-as-artifacts.patch (text/x-diff, 3.4K) download
From e523bd786c034587d6c45079393ba1563cd16eff Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Sat, 26 Feb 2022 19:39:10 -0600
Subject: [PATCH 6/7] cirrus: upload changed html docs as artifacts
This could be done on the client side (cfbot). One advantage of doing
it here is that fewer docs are uploaded - many patches won't upload docs
at all.
https://www.postgresql.org/message-id/flat/20220409021853.GP24419@telsasoft.com
https://www.postgresql.org/message-id/CAB8KJ=i4qmEuopQ+PCSMBzGd4O-Xv0FCnC+q1x7hN9hsdvkBug@mail.gmail.com
https://cirrus-ci.com/task/5396696388599808
ci-os-only: html
---
.cirrus.yml | 24 +++++++++++++++++++++++-
src/tools/ci/copy-changed-docs | 29 +++++++++++++++++++++++++++++
2 files changed, 52 insertions(+), 1 deletion(-)
create mode 100755 src/tools/ci/copy-changed-docs
diff --git a/.cirrus.yml b/.cirrus.ymlindex 123f6957513..a6a246882c9 100644--- a/.cirrus.yml+++ b/.cirrus.yml@@ -27,6 +27,14 @@ env:
TEMP_CONFIG: ${CIRRUS_WORKING_DIR}/src/tools/ci/pg_ci_base.conf
PG_TEST_EXTRA: kerberos ldap ssl
+ # The commit that this branch is rebased on. There's no easy way to find this.+ # This does the right thing for cfbot, which always squishes all patches into a single commit.+ # And does the right thing for any 1-patch commits.+ # Patch series manually submitted to cirrus would benefit from setting this to the+ # number of patches in the series (or directly to the commit the series was rebased on).+ #BASE_COMMIT: HEAD~1+ # For demo purposes:+ BASE_COMMIT: HEAD~11
# What files to preserve in case tests fail
on_failure_ac: &on_failure_ac
@@ -678,6 +686,7 @@ task:
gcc -v
clang -v
export
+ git diff --name-only "$BASE_COMMIT"
ccache_cache:
folder: $CCACHE_DIR
@@ -748,7 +757,7 @@ task:
time make -s -j${BUILD_JOBS} world-bin
###
- # Verify docs can be built+ # Verify docs can be built, and upload changed docs as artifacts
###
# XXX: Only do this if there have been changes in doc/ since last build
docs_build_script: |
@@ -759,6 +768,19 @@ task:
CLANG="ccache clang"
make -s -j${BUILD_JOBS} clean
time make -s -j${BUILD_JOBS} -C doc
+ cp -r doc new-docs++ # Re-build HTML docs from the base commit.+ git checkout "$BASE_COMMIT" -- doc+ make -s -C doc clean+ time make -s -C doc html+ cp -r doc old-docs++ copy_changed_docs_script:+ - src/tools/ci/copy-changed-docs "old-docs" "new-docs" "html_docs"++ html_docs_artifacts:+ paths: ['html_docs/*.html', 'html_docs/*.png', 'html_docs/*.css']
###
# Verify headerscheck / cpluspluscheck succeed
diff --git a/src/tools/ci/copy-changed-docs b/src/tools/ci/copy-changed-docsnew file mode 100755index 00000000000..1c921a8df6f--- /dev/null+++ b/src/tools/ci/copy-changed-docs@@ -0,0 +1,29 @@+#! /bin/sh+# Copy HTML which differ between $old and $new into $outdir+set -e++old=$1+new=$2+outdir=$3++# The index is large and changes often+skippages="bookindex.html"++mkdir "$outdir"+cp "$new"/src/sgml/html/*.css "$new"/src/sgml/html/*.svg "$outdir"++changed=`git diff --no-index --name-only "$old"/src/sgml/html "$new"/src/sgml/html` ||+ [ $? -eq 1 ]++for f in $changed+do+ # Avoid removed files+ [ -f "$f" ] || continue++ echo "$f" |grep -Ew "$skippages" >/dev/null &&+ continue++ cp -v "$f" "$outdir"+done++exit 0--
2.25.1
0007-WIP-ci-meson-allow-showing-only-failed-tests.patch (text/x-diff, 3.7K) download
From e0ef15ae513d62a42887dcbe135e2bc2c477074d Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Fri, 25 Nov 2022 13:57:17 -0600
Subject: [PATCH 7/7] WIP: ci/meson: allow showing only failed tests ..
It's simpler and seems to make more sense to integrate this with
testwrap, rather than to run it after check-world, but only if it
failed, and finding a way to preserve the exit code.
https://www.postgresql.org/message-id/20221114235328.lxdj3puenfhirhqm@awork3.anarazel.de
> It is wasteful to upload thousdands of logfiles to show a single
> failure. That would make our cirrus tasks faster - compressing and
> uploading the logs takes over a minute.
>
> It's also a lot friendlier to show fewer than 8 pages of test folders to
> search through to find the one that failed.
macos
ci-os-only: linux-meson freebsd
---
.cirrus.yml | 13 +++++++------
src/tools/testwrap | 6 ++++++
2 files changed, 13 insertions(+), 6 deletions(-)
diff --git a/.cirrus.yml b/.cirrus.ymlindex a6a246882c9..f64aad64756 100644--- a/.cirrus.yml+++ b/.cirrus.yml@@ -23,6 +23,7 @@ env:
CHECKFLAGS: -Otarget
PROVE_FLAGS: --timer
MTEST_ARGS: --print-errorlogs --no-rebuild -C build
+ PG_FAILED_TESTDIR: ${CIRRUS_WORKING_DIR}/failed.build
PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests
TEMP_CONFIG: ${CIRRUS_WORKING_DIR}/src/tools/ci/pg_ci_base.conf
PG_TEST_EXTRA: kerberos ldap ssl
@@ -48,9 +49,9 @@ on_failure_ac: &on_failure_ac
on_failure_meson: &on_failure_meson
testrun_artifacts:
paths:
- - "build*/testrun/**/*.log"- - "build*/testrun/**/*.diffs"- - "build*/testrun/**/regress_log_*"+ - "failed.build*/**/*.log"+ - "failed.build*/**/*.diffs"+ - "failed.build*/**/regress_log_*"
type: text/plain
# In theory it'd be nice to upload the junit files meson generates, so that
@@ -206,10 +207,10 @@ task:
ulimit -c unlimited
meson test $MTEST_ARGS --quiet --suite setup
export LD_LIBRARY_PATH="$(pwd)/build/tmp_install/usr/local/pgsql/lib/:$LD_LIBRARY_PATH"
- mkdir -p build/testrun+ mkdir -p build/testrun ${PG_FAILED_TESTDIR}
build/tmp_install/usr/local/pgsql/bin/initdb -N build/runningcheck --no-instructions -A trust
echo "include '$(pwd)/src/tools/ci/pg_ci_base.conf'" >> build/runningcheck/postgresql.conf
- build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start+ build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l ${PG_FAILED_TESTDIR}/runningcheck.log start
meson test $MTEST_ARGS --num-processes ${TEST_JOBS} --setup running
build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop
EOF
@@ -397,7 +398,7 @@ task:
test_world_32_script: |
su postgres <<-EOF
ulimit -c unlimited
- PYTHONCOERCECLOCALE=0 LANG=C meson test $MTEST_ARGS -C build-32 --num-processes ${TEST_JOBS}+ PYTHONCOERCECLOCALE=0 LANG=C PG_FAILED_TESTDIR=`pwd`/failed.build-32 meson test $MTEST_ARGS -C build-32 --num-processes ${TEST_JOBS}
EOF
on_failure:
diff --git a/src/tools/testwrap b/src/tools/testwrapindex 7a64fe76a2d..445ac595afc 100755--- a/src/tools/testwrap+++ b/src/tools/testwrap@@ -44,4 +44,10 @@ if sp.returncode == 0:
else:
print('# test failed')
open(os.path.join(testdir, 'test.fail'), 'x')
+ faileddir = os.getenv('PG_FAILED_TESTDIR')+ if faileddir:+ parentdir = os.path.dirname(testdir)+ newdest = os.path.join(faileddir, os.path.basename(parentdir), os.path.basename(testdir))+ shutil.copytree(testdir, newdest)+
sys.exit(sp.returncode)
--
2.25.1
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: pryzby@telsasoft.com, andres@anarazel.de, andrew@dunslane.net, thomas.munro@gmail.com, noah@leadboat.com, michael@paquier.xyz, lubennikovaav@gmail.com, tgl@sss.pgh.pa.us, robertmhaas@gmail.com, melanieplageman@gmail.com, peter.eisentraut@enterprisedb.com, daniel@yesql.se, smilingsamay@gmail.com
Subject: Re: CI and test improvements
In-Reply-To: <20230104234424.GE3109@telsasoft.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox