agora inbox for pgsql-hackers@postgresql.org
help / color / mirror / Atom feedFrom: Justin Pryzby <pryzby@telsasoft.com>
To: Andres Freund <andres@anarazel.de>
Cc: Andrew Dunstan <andrew@dunslane.net>
Cc: Thomas Munro <thomas.munro@gmail.com>
Cc: pgsql-hackers@lists.postgresql.org, Noah Misch <noah@leadboat.com>
Cc: Michael Paquier <michael@paquier.xyz>
Cc: Anastasia Lubennikova <lubennikovaav@gmail.com>
Cc: Tom Lane <tgl@sss.pgh.pa.us>
Cc: Robert Haas <robertmhaas@gmail.com>
Cc: Melanie Plageman <melanieplageman@gmail.com>
Cc: Peter Eisentraut <peter.eisentraut@enterprisedb.com>
Cc: Daniel Gustafsson <daniel@yesql.se>
Cc: samay sharma <smilingsamay@gmail.com>
Subject: Re: CI and test improvements
Date: Tue, 22 Nov 2022 16:57:44 -0600
Message-ID: <20221122225744.GF11463@telsasoft.com> (raw)
In-Reply-To: <20221121224542.p2zapvyvb7objluw@alap3.anarazel.de>
References: <20220528153741.GK19626@telsasoft.com>
<20220828144447.GA21897@telsasoft.com>
<20220828160752.l5l66k3eptokzhzj@awork3.anarazel.de>
<20220828171029.GO2342@telsasoft.com>
<20220828212802.r6eymfffrgr3lxxt@awork3.anarazel.de>
<20220910200542.GX31833@telsasoft.com>
<20221104235412.GE16921@telsasoft.com>
<20221105015946.yrxijqb7h4rqhp6d@awork3.anarazel.de>
<20221113235303.GA26337@telsasoft.com>
<20221121224542.p2zapvyvb7objluw@alap3.anarazel.de>
On Mon, Nov 21, 2022 at 02:45:42PM -0800, Andres Freund wrote:
> > > > + ninja -C build |tee build/meson-logs/build.txt
> > > > + REM Since pipes lose exit status of the preceding command, rerun compilation,
> > > > + REM without the pipe exiting now if it fails, rather than trying to run checks
> > > > + ninja -C build > nul
> > >
> > > This seems mighty grotty :(. but I guess it's quick enough not worry about,
> > > and I can't come up with a better plan.
> > >
> > > It doesn't seem quite right to redirect into meson-logs/ to me, my
> > > interpretation is that that's "meson's namespace". Why not just store it in
> > > build/?
> >
> > I put it there so it'd be included with the build artifacts.
>
> Wouldn't just naming it build-warnings.log suffice? I don't think we
> want to actually upload build.txt - it already is captured.
Originally, I wanted the input and the output to be available as files
and not just in cirrus' web GUI, but maybe that's not important anymore.
I rewrote it again.
> > I'm not sure if this ought to be combined with/before/after your "move
> > compilerwarnings task to meson" patch? (Regarding that patch: I
> > mentioned that it shouldn't use ccache -C, and it should use
> > meson_log_artifacts.)
>
> TBH, I'm not quite sure a separate docs task does really still make
> sense after the SanityCheck task. It's worth building the docs even if
> some flappy test fails, but I don't think we should try to build the
> docs if the code doesn't even compile, in all likelihood a lot more is
> wrong in that case.
It'd be okay either way. I had split it out to 1) isolate the changes
in the "upload changed docs as artifacts" patch; and, 2) so the docs
artifacts are visible in a cfbot link called "Documentation"; and, 3) so
the docs task runs without a dependency on "Linux", since (as you said)
docs/errors are worth showing/reviewing/reporting/addressing separately
from test errors (perhaps similar to compiler warnings...).
I shuffled my branch around and sending now the current "docs" patches,
but I suppose this is waiting on the "convert CompilerWarnings task to
meson" patch.
--
Justin
Attachments:
[text/x-diff] 0001-cirrus-windows-add-compiler_warnings_script.patch (1.6K, ../20221122225744.GF11463@telsasoft.com/2-0001-cirrus-windows-add-compiler_warnings_script.patch)
download | inline diff:
From e7eb22b85dcfe503810ea0f89f48833f04f80d3f Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Wed, 25 May 2022 21:53:22 -0500
Subject: [PATCH 01/10] cirrus/windows: add compiler_warnings_script
I'm not sure how to write this test in windows shell; it's also easy to
write something that doesn't work in posix sh, since windows shell is
interpretting && and ||...
https://www.postgresql.org/message-id/20220212212310.f645c6vw3njkgxka%40alap3.anarazel.de
See also:
8a1ce5e54f6d144e4f8e19af7c767b026ee0c956
https://cirrus-ci.com/task/6241060062494720
https://cirrus-ci.com/task/6496366607204352
ci-os-only: windows
---
.cirrus.yml | 11 ++++++++++-
1 file changed, 10 insertions(+), 1 deletion(-)
diff --git a/.cirrus.yml b/.cirrus.yml
index f31923333ef..6ce4f393e29 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -547,12 +547,21 @@ task:
build_script: |
vcvarsall x64
- ninja -C build
+ ninja -C build |tee build.txt
+ REM Since pipes lose the exit status of the preceding command, rerun the compilation
+ REM without the pipe, exiting now if it fails, to avoid trying to run checks
+ ninja -C build > nul
check_world_script: |
vcvarsall x64
meson test %MTEST_ARGS% --num-processes %TEST_JOBS%
+ # This should be last, so check_world is run even if there are warnings
+ always:
+ compiler_warnings_script:
+ # this avoids using metachars which would be interpretted by the windows shell
+ - sh -c 'if grep ": warning " build.txt; then exit 1; fi; exit 0'
+
on_failure:
<<: *on_failure_meson
crashlog_artifacts:
--
2.25.1
[text/x-diff] 0002-cirrus-macos-switch-to-macos_instance-M1.patch (2.4K, ../20221122225744.GF11463@telsasoft.com/3-0002-cirrus-macos-switch-to-macos_instance-M1.patch)
download | inline diff:
From 79586f7ac46ce48648b150b6ad08987578c730f5 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Fri, 11 Nov 2022 21:14:39 -0600
Subject: [PATCH 02/10] cirrus/macos: switch to "macos_instance" / M1..
This uses an M1 ARM processor rather than intel.
See:
https://cirrus-ci.org/blog/2022/11/08/sunsetting-intel-macos-instances/
https://github.com/justinpryzby/postgres/runs/9446478561
| warning We are sunsetting support for Intel-based macOS instances! Please migrate before January 1st 2023.
ci-os-only: macos
---
.cirrus.yml | 10 ++++++----
src/test/kerberos/t/001_auth.pl | 14 ++++++++++++--
2 files changed, 18 insertions(+), 6 deletions(-)
diff --git a/.cirrus.yml b/.cirrus.yml
index 6ce4f393e29..ed05c4adf87 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -391,7 +391,7 @@ task:
name: macOS - Monterey - Meson
env:
- CPUS: 12 # always get that much for cirrusci macOS instances
+ CPUS: 4 # always get that much for cirrusci macOS instances
BUILD_JOBS: $CPUS
# Test performance regresses noticably when using all cores. 8 seems to
# work OK. See
@@ -412,8 +412,10 @@ task:
depends_on: SanityCheck
only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*(macos|darwin|osx).*'
- osx_instance:
- image: monterey-base
+ macos_instance:
+ image: ghcr.io/cirruslabs/macos-monterey-base:latest
+ #cpus: 4
+ #memory: 8G
sysinfo_script: |
id
@@ -459,7 +461,7 @@ task:
ccache_cache:
folder: $CCACHE_DIR
configure_script: |
- brewpath="/usr/local"
+ brewpath="/opt/homebrew"
PKG_CONFIG_PATH="${brewpath}/lib/pkgconfig:${PKG_CONFIG_PATH}"
for pkg in icu4c krb5 openldap openssl zstd ; do
diff --git a/src/test/kerberos/t/001_auth.pl b/src/test/kerberos/t/001_auth.pl
index a2bc8a5351e..1f6e45b372e 100644
--- a/src/test/kerberos/t/001_auth.pl
+++ b/src/test/kerberos/t/001_auth.pl
@@ -34,8 +34,18 @@ my ($krb5_bin_dir, $krb5_sbin_dir);
if ($^O eq 'darwin')
{
- $krb5_bin_dir = '/usr/local/opt/krb5/bin';
- $krb5_sbin_dir = '/usr/local/opt/krb5/sbin';
+ if (-d "/usr/local/opt/krb5")
+ {
+ # old
+ $krb5_bin_dir = '/usr/local/opt/krb5/bin';
+ $krb5_sbin_dir = '/usr/local/opt/krb5/sbin';
+ }
+ elsif (-d "/opt/homebrew/opt/krb5")
+ {
+ # new
+ $krb5_bin_dir = '/opt/homebrew/opt/krb5/bin';
+ $krb5_sbin_dir = '/opt/homebrew/opt/krb5/sbin';
+ }
}
elsif ($^O eq 'freebsd')
{
--
2.25.1
[text/x-diff] 0003-cirrus-macos-update-to-macos-ventura.patch (937B, ../20221122225744.GF11463@telsasoft.com/4-0003-cirrus-macos-update-to-macos-ventura.patch)
download | inline diff:
From 9d340b707897985d18c4e6f10fb9da41b14e59e9 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Fri, 4 Nov 2022 15:43:56 -0500
Subject: [PATCH 03/10] cirrus/macos: update to macos ventura
ci-os-only: macos
---
.cirrus.yml | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/.cirrus.yml b/.cirrus.yml
index ed05c4adf87..7c6e3691093 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -388,7 +388,7 @@ task:
task:
- name: macOS - Monterey - Meson
+ name: macOS - Ventura - Meson
env:
CPUS: 4 # always get that much for cirrusci macOS instances
@@ -413,7 +413,7 @@ task:
only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*(macos|darwin|osx).*'
macos_instance:
- image: ghcr.io/cirruslabs/macos-monterey-base:latest
+ image: ghcr.io/cirruslabs/macos-ventura-base:latest
#cpus: 4
#memory: 8G
--
2.25.1
[text/x-diff] 0004-cirrus-freebsd-run-with-more-CPUs-RAM-and-do-not-rep.patch (1.8K, ../20221122225744.GF11463@telsasoft.com/5-0004-cirrus-freebsd-run-with-more-CPUs-RAM-and-do-not-rep.patch)
download | inline diff:
From 766f9c2b8114ace0431877cbb1fc8214aeeb8e5b Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Fri, 24 Jun 2022 00:09:12 -0500
Subject: [PATCH 04/10] cirrus/freebsd: run with more CPUs+RAM and do not
repartition
There was some historic problem where tests under freebsd took 8+ minutes (and
before 4a288a37f took 15 minutes).
This reduces test time from 10min to 3min.
4 CPUs 4 tests https://cirrus-ci.com/task/4880240739614720
4 CPUs 6 tests https://cirrus-ci.com/task/4664440120410112 https://cirrus-ci.com/task/4586784884523008
4 CPUs 8 tests https://cirrus-ci.com/task/5001995491737600
6 CPUs https://cirrus-ci.com/task/6678321684545536
8 CPUs https://cirrus-ci.com/task/6264854121021440
See also:
https://www.postgresql.org/message-id/flat/20220310033347.hgxk4pyarzq4hxwp@alap3.anarazel.de#f36c0b17e33e31e7925e7e5812998686
8 jobs 7min https://cirrus-ci.com/task/6186376667332608
//-os-only: freebsd
---
.cirrus.yml | 10 +++-------
1 file changed, 3 insertions(+), 7 deletions(-)
diff --git a/.cirrus.yml b/.cirrus.yml
index 7c6e3691093..6aaaf62d6a9 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -131,11 +131,9 @@ task:
name: FreeBSD - 13 - Meson
env:
- # FreeBSD on GCP is slow when running with larger number of CPUS /
- # jobs. Using one more job than cpus seems to work best.
- CPUS: 2
- BUILD_JOBS: 3
- TEST_JOBS: 3
+ CPUS: 4
+ BUILD_JOBS: 4
+ TEST_JOBS: 6
CCACHE_DIR: /tmp/ccache_dir
CPPFLAGS: -DRELCACHE_FORCE_RELEASE -DCOPY_PARSE_PLAN_TREES -DWRITE_READ_PARSE_PLAN_TREES -DRAW_EXPRESSION_COVERAGE_TEST
@@ -160,8 +158,6 @@ task:
ccache_cache:
folder: $CCACHE_DIR
- # Workaround around performance issues due to 32KB block size
- repartition_script: src/tools/ci/gcp_freebsd_repartition.sh
create_user_script: |
pw useradd postgres
chown -R postgres:postgres .
--
2.25.1
[text/x-diff] 0005-cirrus-clean-up-typos.patch (1.9K, ../20221122225744.GF11463@telsasoft.com/6-0005-cirrus-clean-up-typos.patch)
download | inline diff:
From b36213a7801d354e8f47189d19b7c9166b1ad22b Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Sun, 13 Nov 2022 14:03:27 -0600
Subject: [PATCH 05/10] cirrus: clean up typos
---
.cirrus.yml | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/.cirrus.yml b/.cirrus.yml
index 6aaaf62d6a9..3eda0268161 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -47,7 +47,7 @@ on_failure_meson: &on_failure_meson
# In theory it'd be nice to upload the junit files meson generates, so that
# cirrus will nicely annotate the commit. Unfortunately the files don't
- # contain identifieable file + line numbers right now, so the annotations
+ # contain identifiable file + line numbers right now, so the annotations
# don't end up useful. We could probably improve on that with a some custom
# conversion script, but ...
meson_log_artifacts:
@@ -74,7 +74,7 @@ task:
# no options enabled, should be small
CCACHE_MAXSIZE: "150M"
- # Container starts up quickly, but is slower at runtime, particularly for
+ # Containers start up quickly, but is slower at runtime, particularly for
# tests. Good for the briefly running sanity check.
container:
image: $CONTAINER_REPO/linux_debian_bullseye_ci:latest
@@ -590,7 +590,7 @@ task:
CCACHE_MAXSIZE: "500M"
CCACHE_SLOPPINESS: pch_defines,time_macros
CCACHE_DEPEND: 1
- # for some reason mingw plpython cannot find it's installation without this
+ # for some reason mingw plpython cannot find its installation without this
PYTHONHOME: C:/msys64/ucrt64
# prevents MSYS bash from resetting error mode
MSYS: winjitdebug
@@ -645,7 +645,7 @@ task:
# Use larger ccache cache, as this task compiles with multiple compilers /
# flag combinations
- CCACHE_MAXSIZE: "1GB"
+ CCACHE_MAXSIZE: "1G"
CCACHE_DIR: "/tmp/ccache_dir"
LINUX_CONFIGURE_FEATURES: *LINUX_CONFIGURE_FEATURES
--
2.25.1
[text/x-diff] 0006-cirrus-build-docs-as-a-separate-task.patch (2.4K, ../20221122225744.GF11463@telsasoft.com/7-0006-cirrus-build-docs-as-a-separate-task.patch)
download | inline diff:
From 7133474e6280538439f43f799726275ce69ee2cd Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Sat, 26 Feb 2022 19:34:35 -0600
Subject: [PATCH 06/10] cirrus: build docs as a separate task..
This will run the doc build if any docs have changed, even if Linux
fails, to allow catch doc build failures.
This automatically shows up as a separate "column" on cfbot.
Also, in the future, this will hopefully upload each patch's changed HTML docs
as an artifact, for easy review.
XXX: it ought to be possible to skip (re)building docs if they haven't
changed since the last build. But that doesn't appear to be working.
https://cirrus-ci.com/task/4523764229603328
skip: "!changesInclude('.cirrus.yml', 'doc/**')"
ci-os-only: html
---
.cirrus.yml | 48 ++++++++++++++++++++++++++++++++++--------------
1 file changed, 34 insertions(+), 14 deletions(-)
diff --git a/.cirrus.yml b/.cirrus.yml
index 3eda0268161..5cb18032779 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -736,20 +736,6 @@ task:
make -s -j${BUILD_JOBS} clean
time make -s -j${BUILD_JOBS} world-bin
- ###
- # Verify docs can be built
- ###
- # XXX: Only do this if there have been changes in doc/ since last build
- always:
- docs_build_script: |
- time ./configure \
- --cache gcc.cache \
- CC="ccache gcc" \
- CXX="ccache g++" \
- CLANG="ccache clang"
- make -s -j${BUILD_JOBS} clean
- time make -s -j${BUILD_JOBS} -C doc
-
###
# Verify headerscheck / cpluspluscheck succeed
#
@@ -773,3 +759,37 @@ task:
always:
upload_caches: ccache
+
+
+###
+# Verify docs can be built
+# changesInclude() will skip this task if none of the commits since
+# CIRRUS_LAST_GREEN_CHANGE touched any relevant files. The comparison appears
+# to be like "git log a..b -- ./file", not "git diff a..b -- ./file"
+###
+
+task:
+ name: Documentation
+
+ env:
+ CPUS: 1
+ BUILD_JOBS: 1
+
+ only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*(docs|html).*'
+
+ container:
+ image: $CONTAINER_REPO/linux_debian_bullseye_ci:latest
+ cpu: $CPUS
+ memory: 2G
+
+ # Exercise HTML and other docs:
+ ninja_docs_build_script: |
+ mkdir build-ninja
+ cd build-ninja
+ time meson setup
+ time ninja docs
+
+ # Again, with autoconf
+ docs_build_script: |
+ time ./configure
+ make -s -j${BUILD_JOBS} -C doc
--
2.25.1
[text/x-diff] 0007-cirrus-warnings-use-a-single-common-always-block.patch (3.2K, ../20221122225744.GF11463@telsasoft.com/8-0007-cirrus-warnings-use-a-single-common-always-block.patch)
download | inline diff:
From 0686c222fdf36bd2b844cbe9c86d522dfc030fc8 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Tue, 19 Jul 2022 12:38:45 -0500
Subject: [PATCH 07/10] cirrus/warnings: use a single/common 'always' block
ci-os-only: warnings
---
.cirrus.yml | 40 +++++++++++++++++-----------------------
1 file changed, 17 insertions(+), 23 deletions(-)
diff --git a/.cirrus.yml b/.cirrus.yml
index 5cb18032779..0bc7c5e3796 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -664,9 +664,6 @@ task:
clang -v
export
- ccache_cache:
- folder: $CCACHE_DIR
-
setup_additional_packages_script: |
#apt-get update
#DEBIAN_FRONTEND=noninteractive apt-get -y install ...
@@ -681,8 +678,11 @@ task:
# different compilers to build with different combinations of dtrace on/off
# and cassert on/off.
- # gcc, cassert off, dtrace on
always:
+ ccache_cache:
+ folder: $CCACHE_DIR
+
+ # gcc, cassert off, dtrace on
gcc_warning_script: |
time ./configure \
--cache gcc.cache \
@@ -692,8 +692,7 @@ task:
make -s -j${BUILD_JOBS} clean
time make -s -j${BUILD_JOBS} world-bin
- # gcc, cassert on, dtrace off
- always:
+ # gcc, cassert on, dtrace off
gcc_a_warning_script: |
time ./configure \
--cache gcc.cache \
@@ -703,8 +702,7 @@ task:
make -s -j${BUILD_JOBS} clean
time make -s -j${BUILD_JOBS} world-bin
- # clang, cassert off, dtrace off
- always:
+ # clang, cassert off, dtrace off
clang_warning_script: |
time ./configure \
--cache clang.cache \
@@ -713,8 +711,7 @@ task:
make -s -j${BUILD_JOBS} clean
time make -s -j${BUILD_JOBS} world-bin
- # clang, cassert on, dtrace on
- always:
+ # clang, cassert on, dtrace on
clang_a_warning_script: |
time ./configure \
--cache clang.cache \
@@ -725,8 +722,7 @@ task:
make -s -j${BUILD_JOBS} clean
time make -s -j${BUILD_JOBS} world-bin
- # cross-compile to windows
- always:
+ # cross-compile to windows
mingw_cross_warning_script: |
time ./configure \
--host=x86_64-w64-mingw32 \
@@ -736,16 +732,15 @@ task:
make -s -j${BUILD_JOBS} clean
time make -s -j${BUILD_JOBS} world-bin
- ###
- # Verify headerscheck / cpluspluscheck succeed
- #
- # - Don't use ccache, the files are uncacheable, polluting ccache's
- # cache
- # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose
- # - XXX have to disable ICU to avoid errors:
- # https://postgr.es/m/20220323002024.f2g6tivduzrktgfa%40alap3.anarazel.de
- ###
- always:
+ ###
+ # Verify headerscheck / cpluspluscheck succeed
+ #
+ # - Don't use ccache, the files are uncacheable, polluting ccache's
+ # cache
+ # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose
+ # - XXX have to disable ICU to avoid errors:
+ # https://postgr.es/m/20220323002024.f2g6tivduzrktgfa%40alap3.anarazel.de
+ ###
headers_headerscheck_script: |
time ./configure \
${LINUX_CONFIGURE_FEATURES} \
@@ -757,7 +752,6 @@ task:
headers_cpluspluscheck_script: |
time make -s cpluspluscheck EXTRAFLAGS='-fmax-errors=10'
- always:
upload_caches: ccache
--
2.25.1
[text/x-diff] 0008-cirrus-upload-changed-html-docs-as-artifacts.patch (3.9K, ../20221122225744.GF11463@telsasoft.com/9-0008-cirrus-upload-changed-html-docs-as-artifacts.patch)
download | inline diff:
From 994c0df0b98fab4f10ac987f13302e0eb3aa07a9 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Sat, 26 Feb 2022 19:39:10 -0600
Subject: [PATCH 08/10] cirrus: upload changed html docs as artifacts
This could be done on the client side (cfbot). One advantage of doing it here
is that fewer docs are uploaded - many patches won't upload docs at all.
https://www.postgresql.org/message-id/flat/20220409021853.GP24419@telsasoft.com
https://www.postgresql.org/message-id/CAB8KJ=i4qmEuopQ+PCSMBzGd4O-Xv0FCnC+q1x7hN9hsdvkBug@mail.gmail.com
https://cirrus-ci.com/task/5396696388599808
ci-os-only: html
---
.cirrus.yml | 31 ++++++++++++++++++++++++++++---
src/tools/ci/copy-changed-docs | 29 +++++++++++++++++++++++++++++
2 files changed, 57 insertions(+), 3 deletions(-)
create mode 100755 src/tools/ci/copy-changed-docs
diff --git a/.cirrus.yml b/.cirrus.yml
index 0bc7c5e3796..2f2c09027d8 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -27,6 +27,14 @@ env:
TEMP_CONFIG: ${CIRRUS_WORKING_DIR}/src/tools/ci/pg_ci_base.conf
PG_TEST_EXTRA: kerberos ldap ssl
+ # The commit that this branch is rebased on. There's no easy way to find this.
+ # This does the right thing for cfbot, which always squishes all patches into a single commit.
+ # And does the right thing for any 1-patch commits.
+ # Patch series manually submitted to cirrus would benefit from setting this to the
+ # number of patches in the series (or directly to the commit the series was rebased on).
+ #BASE_COMMIT: HEAD~1
+ # For demo purposes:
+ BASE_COMMIT: 3d14e171e9e2236139e8976f3309a588bcc8683b
# What files to preserve in case tests fail
on_failure_ac: &on_failure_ac
@@ -756,7 +764,7 @@ task:
###
-# Verify docs can be built
+# Verify docs can be built, and upload changed docs as artifacts
# changesInclude() will skip this task if none of the commits since
# CIRRUS_LAST_GREEN_CHANGE touched any relevant files. The comparison appears
# to be like "git log a..b -- ./file", not "git diff a..b -- ./file"
@@ -764,18 +772,21 @@ task:
task:
name: Documentation
+ only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*(docs|html).*'
+ #skip: "!changesInclude('.cirrus.yml', 'doc/**', 'src/tools/ci/copy-changed-docs')"
env:
CPUS: 1
BUILD_JOBS: 1
- only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*(docs|html).*'
-
container:
image: $CONTAINER_REPO/linux_debian_bullseye_ci:latest
cpu: $CPUS
memory: 2G
+ sysinfo_script: |
+ git diff --name-only "$BASE_COMMIT"
+
# Exercise HTML and other docs:
ninja_docs_build_script: |
mkdir build-ninja
@@ -787,3 +798,17 @@ task:
docs_build_script: |
time ./configure
make -s -j${BUILD_JOBS} -C doc
+ cp -r doc new-docs
+
+ # Build HTML docs from the base commit.
+ git checkout "$BASE_COMMIT" -- doc
+ #html-stamp
+ make -s -C doc clean
+ make -s -C doc html
+ cp -r doc old-docs
+
+ copy_changed_docs_script: |
+ src/tools/ci/copy-changed-docs "old-docs" "new-docs" "html_docs"
+
+ html_docs_artifacts:
+ paths: ['html_docs/*.html', 'html_docs/*.png', 'html_docs/*.css']
diff --git a/src/tools/ci/copy-changed-docs b/src/tools/ci/copy-changed-docs
new file mode 100755
index 00000000000..1c921a8df6f
--- /dev/null
+++ b/src/tools/ci/copy-changed-docs
@@ -0,0 +1,29 @@
+#! /bin/sh
+# Copy HTML which differ between $old and $new into $outdir
+set -e
+
+old=$1
+new=$2
+outdir=$3
+
+# The index is large and changes often
+skippages="bookindex.html"
+
+mkdir "$outdir"
+cp "$new"/src/sgml/html/*.css "$new"/src/sgml/html/*.svg "$outdir"
+
+changed=`git diff --no-index --name-only "$old"/src/sgml/html "$new"/src/sgml/html` ||
+ [ $? -eq 1 ]
+
+for f in $changed
+do
+ # Avoid removed files
+ [ -f "$f" ] || continue
+
+ echo "$f" |grep -Ew "$skippages" >/dev/null &&
+ continue
+
+ cp -v "$f" "$outdir"
+done
+
+exit 0
--
2.25.1
[text/x-diff] 0009-WIP-show-changed-docs-with-meson.patch (2.1K, ../20221122225744.GF11463@telsasoft.com/10-0009-WIP-show-changed-docs-with-meson.patch)
download | inline diff:
From 3bd2db1aba704bb2fd1d8ebd5cec349770ee5e21 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Wed, 19 Oct 2022 20:33:28 -0500
Subject: [PATCH 09/10] WIP: show changed docs with meson
ci-os-only: html docs
---
.cirrus.yml | 27 ++++++++++-----------------
src/tools/ci/copy-changed-docs | 3 ++-
2 files changed, 12 insertions(+), 18 deletions(-)
diff --git a/.cirrus.yml b/.cirrus.yml
index 2f2c09027d8..338c6c2be16 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -776,8 +776,7 @@ task:
#skip: "!changesInclude('.cirrus.yml', 'doc/**', 'src/tools/ci/copy-changed-docs')"
env:
- CPUS: 1
- BUILD_JOBS: 1
+ CPUS: 2
container:
image: $CONTAINER_REPO/linux_debian_bullseye_ci:latest
@@ -793,22 +792,16 @@ task:
cd build-ninja
time meson setup
time ninja docs
+ cp -r doc ../new-docs
- # Again, with autoconf
- docs_build_script: |
- time ./configure
- make -s -j${BUILD_JOBS} -C doc
- cp -r doc new-docs
-
- # Build HTML docs from the base commit.
- git checkout "$BASE_COMMIT" -- doc
- #html-stamp
- make -s -C doc clean
- make -s -C doc html
- cp -r doc old-docs
-
- copy_changed_docs_script: |
- src/tools/ci/copy-changed-docs "old-docs" "new-docs" "html_docs"
+ # Re-build docs from the base commit.
+ git checkout "$BASE_COMMIT" -- ../doc
+ ninja clean
+ time ninja doc/src/sgml/html
+ cp -r doc ../old-docs
+
+ copy_changed_docs_script:
+ - src/tools/ci/copy-changed-docs "old-docs" "new-docs" "html_docs"
html_docs_artifacts:
paths: ['html_docs/*.html', 'html_docs/*.png', 'html_docs/*.css']
diff --git a/src/tools/ci/copy-changed-docs b/src/tools/ci/copy-changed-docs
index 1c921a8df6f..598e639f695 100755
--- a/src/tools/ci/copy-changed-docs
+++ b/src/tools/ci/copy-changed-docs
@@ -10,7 +10,8 @@ outdir=$3
skippages="bookindex.html"
mkdir "$outdir"
-cp "$new"/src/sgml/html/*.css "$new"/src/sgml/html/*.svg "$outdir"
+cp doc/src/sgml/*.css "$outdir"
+cp doc/src/sgml/images/*.svg "$outdir"
changed=`git diff --no-index --name-only "$old"/src/sgml/html "$new"/src/sgml/html` ||
[ $? -eq 1 ]
--
2.25.1
[text/x-diff] 0010-html-index-file.patch (3.2K, ../20221122225744.GF11463@telsasoft.com/11-0010-html-index-file.patch)
download | inline diff:
From 8086a65429b81fbbb2886f5bfa4495ff019b59bf Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Mon, 28 Feb 2022 23:18:19 -0600
Subject: [PATCH 10/10] +html index file
This allows linking to the artifacts from the last successful build, which
itself allows *not* rebuilding when sources haven't changed.
//freebsd
//-os-only: html,
---
.cirrus.yml | 2 +-
src/tools/ci/copy-changed-docs | 42 ++++++++++++++++++++++++++++++++--
2 files changed, 41 insertions(+), 3 deletions(-)
diff --git a/.cirrus.yml b/.cirrus.yml
index 338c6c2be16..5e1612b583e 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -773,7 +773,7 @@ task:
task:
name: Documentation
only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*(docs|html).*'
- #skip: "!changesInclude('.cirrus.yml', 'doc/**', 'src/tools/ci/copy-changed-docs')"
+ skip: "!changesInclude('.cirrus.yml', 'doc/**', 'src/tools/ci/copy-changed-docs')"
env:
CPUS: 2
diff --git a/src/tools/ci/copy-changed-docs b/src/tools/ci/copy-changed-docs
index 598e639f695..c60428e1ae1 100755
--- a/src/tools/ci/copy-changed-docs
+++ b/src/tools/ci/copy-changed-docs
@@ -5,6 +5,10 @@ set -e
old=$1
new=$2
outdir=$3
+branch=$CIRRUS_BRANCH
+
+# The index is large and changes often
+skippages="bookindex.html"
# The index is large and changes often
skippages="bookindex.html"
@@ -13,6 +17,19 @@ mkdir "$outdir"
cp doc/src/sgml/*.css "$outdir"
cp doc/src/sgml/images/*.svg "$outdir"
+# The index is useful to allow a static link (not specific to a cirrus run) to the artifacts for the most-recent, successful CI run for a branch
+branchurl=https://api.cirrus-ci.com/v1/artifact/github/$CIRRUS_REPO_FULL_NAME/$CIRRUS_TASK_NAME/html_docs/html_docs/00-index.html?branch=$branch
+
+index="$outdir/00-index.html"
+cat >"$index" <<EOF
+<html>
+<head><title>Index of docs changed since: $BASE_COMMIT</title></head>
+<body>
+<!-- A link to documentation for the most recent successful build: $branchurl -->
+<h1>Index of docs changed since: $BASE_COMMIT</h1>
+<ul>
+EOF
+
changed=`git diff --no-index --name-only "$old"/src/sgml/html "$new"/src/sgml/html` ||
[ $? -eq 1 ]
@@ -24,7 +41,28 @@ do
echo "$f" |grep -Ew "$skippages" >/dev/null &&
continue
- cp -v "$f" "$outdir"
-done
+ cp -v "$f" "$outdir" >&2
+ fn=${f##*/}
+ # ?branch=... is needed when accessing the artifacts for the static link for the branch
+ # It's not needed and ignored if accessing artifacts for *this* CI run
+ echo "<li><a href='$fn?branch=$branch'>$fn</a>"
+done >>"$index"
+
+github=https://github.com/$CIRRUS_REPO_FULL_NAME/commit
+cirrus=https://cirrus-ci.com/build
+
+cat >>"$index" <<EOF
+</ul>
+<hr>
+<code>
+<br>This file was written on: `date --rfc-822 --utc`
+<br>CIRRUS_CHANGE_TITLE: $CIRRUS_CHANGE_TITLE
+<br>CIRRUS_CHANGE_IN_REPO: <a href="$github/$CIRRUS_CHANGE_IN_REPO">$CIRRUS_CHANGE_IN_REPO</a>
+<br>CIRRUS_BUILD_ID: <a href="$cirrus/$CIRRUS_BUILD_ID">$CIRRUS_BUILD_ID</a>
+<br>CIRRUS_LAST_GREEN_CHANGE: <a href="$github/$CIRRUS_LAST_GREEN_CHANGE">$CIRRUS_LAST_GREEN_CHANGE</a>
+<br>CIRRUS_LAST_GREEN_BUILD_ID: <a href="$cirrus/$CIRRUS_LAST_GREEN_BUILD_ID">$CIRRUS_LAST_GREEN_BUILD_ID</a>
+</code>
+</body></html>
+EOF
exit 0
--
2.25.1
view thread (76+ messages) latest in thread
Message-ID: <20221122225744.GF11463@telsasoft.com>
Permalink: ../20221122225744.GF11463@telsasoft.com/
Also on: postgresql.org/message-id/20221122225744.GF11463@telsasoft.com
reply
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: pryzby@telsasoft.com, andres@anarazel.de, andrew@dunslane.net, thomas.munro@gmail.com, noah@leadboat.com, michael@paquier.xyz, lubennikovaav@gmail.com, tgl@sss.pgh.pa.us, robertmhaas@gmail.com, melanieplageman@gmail.com, peter.eisentraut@enterprisedb.com, daniel@yesql.se, smilingsamay@gmail.com
Subject: Re: CI and test improvements
In-Reply-To: <20221122225744.GF11463@telsasoft.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox