public inbox for [email protected]help / color / mirror / Atom feed
[PATCH v45 7/7] Exclude pg_stat directory from base backup 3+ messages / 3 participants [nested] [flat]
* [PATCH v45 7/7] Exclude pg_stat directory from base backup @ 2020-09-29 14:19 Kyotaro Horiguchi <[email protected]> 0 siblings, 0 replies; 3+ messages in thread From: Kyotaro Horiguchi @ 2020-09-29 14:19 UTC (permalink / raw) basebackup sends the content of pg_stat directory, which is doomed to be removed at startup from the backup. Now that pg_stat_statements saves a temporary file into the directory, let exclude pg_stat directory from a base backup. --- src/backend/replication/basebackup.c | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/src/backend/replication/basebackup.c b/src/backend/replication/basebackup.c index d2c3064678..25677c5c6e 100644 --- a/src/backend/replication/basebackup.c +++ b/src/backend/replication/basebackup.c @@ -149,6 +149,13 @@ struct exclude_list_item */ static const char *const excludeDirContents[] = { + /* + * Skip statistics files. PGSTAT_STAT_PERMANENT_DIRECTORY must be skipped + * because the files in the directory will be removed at startup from the + * backup. + */ + PGSTAT_STAT_PERMANENT_DIRECTORY, + /* * It is generally not useful to backup the contents of this directory * even if the intention is to restore to another primary. See backup.sgml -- 2.27.0 ----Next_Part(Fri_Jan__8_10_24_34_2021_185)---- ^ permalink raw reply [nested|flat] 3+ messages in thread
* Re: CI and test improvements @ 2023-02-03 14:26 Justin Pryzby <[email protected]> 0 siblings, 1 reply; 3+ messages in thread From: Justin Pryzby @ 2023-02-03 14:26 UTC (permalink / raw) To: Thomas Munro <[email protected]>; +Cc: Tom Lane <[email protected]>; Andres Freund <[email protected]>; Andrew Dunstan <[email protected]>; [email protected]; Noah Misch <[email protected]>; Michael Paquier <[email protected]>; Anastasia Lubennikova <[email protected]>; Robert Haas <[email protected]>; Melanie Plageman <[email protected]>; Peter Eisentraut <[email protected]>; Daniel Gustafsson <[email protected]>; samay sharma <[email protected]> rebased, and re-including a patch to show code coverage of changed files. a5b3e50d922 cirrus/windows: add compiler_warnings_script 4c98dcb0e03 cirrus/freebsd: run with more CPUs+RAM and do not repartition aaeef938ed4 cirrus/freebsd: define ENFORCE_REGRESSION_TEST_NAME_RESTRICTIONS 9baf41674ad pg_upgrade: tap test: exercise --link and --clone 7e09035f588 WIP: ci/meson: allow showing only failed tests .. e4534821ef5 cirrus/ccache: use G rather than GB suffix.. 185d1c3ed13 cirrus: code coverage 5dace84a038 cirrus: upload changed html docs as artifacts 852360330ef +html index file Attachments: [text/x-diff] 0001-cirrus-windows-add-compiler_warnings_script.patch (1.6K, ../../[email protected]/2-0001-cirrus-windows-add-compiler_warnings_script.patch) download | inline diff: From a5b3e50d922a6dcff22feabbcae741b23b2347c6 Mon Sep 17 00:00:00 2001 From: Justin Pryzby <[email protected]> Date: Wed, 25 May 2022 21:53:22 -0500 Subject: [PATCH 1/9] cirrus/windows: add compiler_warnings_script I'm not sure how to write this test in windows shell; it's also easy to write something that doesn't work in posix sh, since windows shell is interpretting && and ||... https://www.postgresql.org/message-id/20220212212310.f645c6vw3njkgxka%40alap3.anarazel.de See also: 8a1ce5e54f6d144e4f8e19af7c767b026ee0c956 https://cirrus-ci.com/task/6241060062494720 https://cirrus-ci.com/task/6496366607204352 ci-os-only: windows --- .cirrus.yml | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/.cirrus.yml b/.cirrus.yml index 1204824d2eb..eefc5c21fe6 100644 --- a/.cirrus.yml +++ b/.cirrus.yml @@ -566,12 +566,21 @@ task: build_script: | vcvarsall x64 - ninja -C build + ninja -C build |tee build.txt + REM Since pipes lose the exit status of the preceding command, rerun the compilation + REM without the pipe, exiting now if it fails, to avoid trying to run checks + ninja -C build > nul check_world_script: | vcvarsall x64 meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + # This should be last, so check_world is run even if there are warnings + always: + compiler_warnings_script: + # this avoids using metachars which would be interpretted by the windows shell + - sh -c 'if grep ": warning " build.txt; then exit 1; fi; exit 0' + on_failure: <<: *on_failure_meson crashlog_artifacts: -- 2.25.1 [text/x-diff] 0002-cirrus-freebsd-run-with-more-CPUs-RAM-and-do-not-rep.patch (2.2K, ../../[email protected]/3-0002-cirrus-freebsd-run-with-more-CPUs-RAM-and-do-not-rep.patch) download | inline diff: From 4c98dcb0e033ca12c6a6093c5e94e4231756fa4d Mon Sep 17 00:00:00 2001 From: Justin Pryzby <[email protected]> Date: Fri, 24 Jun 2022 00:09:12 -0500 Subject: [PATCH 2/9] cirrus/freebsd: run with more CPUs+RAM and do not repartition There was some historic problem where tests under freebsd took 8+ minutes (and before 4a288a37f took 15 minutes). This reduces test time from 10min to 3min. 4 CPUs 4 tests https://cirrus-ci.com/task/4880240739614720 4 CPUs 6 tests https://cirrus-ci.com/task/4664440120410112 https://cirrus-ci.com/task/4586784884523008 4 CPUs 8 tests https://cirrus-ci.com/task/5001995491737600 6 CPUs https://cirrus-ci.com/task/6678321684545536 8 CPUs https://cirrus-ci.com/task/6264854121021440 See also: https://www.postgresql.org/message-id/flat/[email protected]#f36c0b17e33e31e7925e7e5812998686 8 jobs 7min https://cirrus-ci.com/task/6186376667332608 //-os-only: freebsd --- .cirrus.yml | 13 ++++--------- 1 file changed, 4 insertions(+), 9 deletions(-) diff --git a/.cirrus.yml b/.cirrus.yml index eefc5c21fe6..0c12ca04fd9 100644 --- a/.cirrus.yml +++ b/.cirrus.yml @@ -131,11 +131,9 @@ task: name: FreeBSD - 13 - Meson env: - # FreeBSD on GCP is slow when running with larger number of CPUS / - # jobs. Using one more job than cpus seems to work best. - CPUS: 2 - BUILD_JOBS: 3 - TEST_JOBS: 3 + CPUS: 4 + BUILD_JOBS: 4 + TEST_JOBS: 6 CCACHE_DIR: /tmp/ccache_dir CPPFLAGS: -DRELCACHE_FORCE_RELEASE -DCOPY_PARSE_PLAN_TREES -DWRITE_READ_PARSE_PLAN_TREES -DRAW_EXPRESSION_COVERAGE_TEST @@ -160,8 +158,6 @@ task: ccache_cache: folder: $CCACHE_DIR - # Work around performance issues due to 32KB block size - repartition_script: src/tools/ci/gcp_freebsd_repartition.sh create_user_script: | pw useradd postgres chown -R postgres:postgres . @@ -175,8 +171,7 @@ task: #pkg install -y ... # NB: Intentionally build without -Dllvm. The freebsd image size is already - # large enough to make VM startup slow, and even without llvm freebsd - # already takes longer than other platforms except for windows. + # large enough to make VM startup slow configure_script: | su postgres <<-EOF meson setup \ -- 2.25.1 [text/x-diff] 0003-cirrus-freebsd-define-ENFORCE_REGRESSION_TEST_NAME_R.patch (934B, ../../[email protected]/4-0003-cirrus-freebsd-define-ENFORCE_REGRESSION_TEST_NAME_R.patch) download | inline diff: From aaeef938ed4a1b604c41d3973a49d790ad58af20 Mon Sep 17 00:00:00 2001 From: Justin Pryzby <[email protected]> Date: Fri, 9 Dec 2022 15:32:50 -0600 Subject: [PATCH 3/9] cirrus/freebsd: define ENFORCE_REGRESSION_TEST_NAME_RESTRICTIONS See also: 54100f5c6052404f68de9ce7310ceb61f1c291f8 ci-os-only: freebsd --- .cirrus.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.cirrus.yml b/.cirrus.yml index 0c12ca04fd9..6186505ccd7 100644 --- a/.cirrus.yml +++ b/.cirrus.yml @@ -136,7 +136,7 @@ task: TEST_JOBS: 6 CCACHE_DIR: /tmp/ccache_dir - CPPFLAGS: -DRELCACHE_FORCE_RELEASE -DCOPY_PARSE_PLAN_TREES -DWRITE_READ_PARSE_PLAN_TREES -DRAW_EXPRESSION_COVERAGE_TEST + CPPFLAGS: -DRELCACHE_FORCE_RELEASE -DCOPY_PARSE_PLAN_TREES -DWRITE_READ_PARSE_PLAN_TREES -DRAW_EXPRESSION_COVERAGE_TEST -DENFORCE_REGRESSION_TEST_NAME_RESTRICTIONS CFLAGS: -Og -ggdb depends_on: SanityCheck -- 2.25.1 [text/x-diff] 0004-pg_upgrade-tap-test-exercise-link-and-clone.patch (1.5K, ../../[email protected]/5-0004-pg_upgrade-tap-test-exercise-link-and-clone.patch) download | inline diff: From 9baf41674ad28d68f2c657363079643955252fee Mon Sep 17 00:00:00 2001 From: Justin Pryzby <[email protected]> Date: Sat, 30 Jul 2022 19:25:51 -0500 Subject: [PATCH 4/9] pg_upgrade: tap test: exercise --link and --clone This both increases code coverage and accelerates tests. See also: b059a2409faf5833b3ba7792e247d6466c9e8090 linux, macos, //-os-only: freebsd --- .cirrus.yml | 3 +++ 1 file changed, 3 insertions(+) diff --git a/.cirrus.yml b/.cirrus.yml index 6186505ccd7..a8a514ed178 100644 --- a/.cirrus.yml +++ b/.cirrus.yml @@ -138,6 +138,7 @@ task: CCACHE_DIR: /tmp/ccache_dir CPPFLAGS: -DRELCACHE_FORCE_RELEASE -DCOPY_PARSE_PLAN_TREES -DWRITE_READ_PARSE_PLAN_TREES -DRAW_EXPRESSION_COVERAGE_TEST -DENFORCE_REGRESSION_TEST_NAME_RESTRICTIONS CFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --link depends_on: SanityCheck only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*freebsd.*' @@ -269,6 +270,7 @@ task: LDFLAGS: $SANITIZER_FLAGS CC: ccache gcc CXX: ccache g++ + PG_TEST_PG_UPGRADE_MODE: --link LINUX_CONFIGURE_FEATURES: *LINUX_CONFIGURE_FEATURES LINUX_MESON_FEATURES: *LINUX_MESON_FEATURES @@ -419,6 +421,7 @@ task: CPPFLAGS: -DRANDOMIZE_ALLOCATED_MEMORY CFLAGS: -Og -ggdb CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone depends_on: SanityCheck only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*(macos|darwin|osx).*' -- 2.25.1 [text/x-diff] 0005-WIP-ci-meson-allow-showing-only-failed-tests.patch (3.7K, ../../[email protected]/6-0005-WIP-ci-meson-allow-showing-only-failed-tests.patch) download | inline diff: From 7e09035f5882a0262fe339eb305e64f3bbaa2bb9 Mon Sep 17 00:00:00 2001 From: Justin Pryzby <[email protected]> Date: Fri, 25 Nov 2022 13:57:17 -0600 Subject: [PATCH 5/9] WIP: ci/meson: allow showing only failed tests .. It's simpler and seems to make more sense to integrate this with testwrap, rather than to run it after check-world, but only if it failed, and finding a way to preserve the exit code. https://www.postgresql.org/message-id/[email protected] > It is wasteful to upload thousdands of logfiles to show a single > failure. That would make our cirrus tasks faster - compressing and > uploading the logs takes over a minute. > > It's also a lot friendlier to show fewer than 8 pages of test folders to > search through to find the one that failed. macos ci-os-only: linux-meson freebsd --- .cirrus.yml | 13 +++++++------ src/tools/testwrap | 6 ++++++ 2 files changed, 13 insertions(+), 6 deletions(-) diff --git a/.cirrus.yml b/.cirrus.yml index a8a514ed178..fd5585e5818 100644 --- a/.cirrus.yml +++ b/.cirrus.yml @@ -23,6 +23,7 @@ env: CHECKFLAGS: -Otarget PROVE_FLAGS: --timer MTEST_ARGS: --print-errorlogs --no-rebuild -C build + PG_FAILED_TESTDIR: ${CIRRUS_WORKING_DIR}/failed.build PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${CIRRUS_WORKING_DIR}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl @@ -40,9 +41,9 @@ on_failure_ac: &on_failure_ac on_failure_meson: &on_failure_meson testrun_artifacts: paths: - - "build*/testrun/**/*.log" - - "build*/testrun/**/*.diffs" - - "build*/testrun/**/regress_log_*" + - "failed.build*/**/*.log" + - "failed.build*/**/*.diffs" + - "failed.build*/**/regress_log_*" type: text/plain # In theory it'd be nice to upload the junit files meson generates, so that @@ -198,10 +199,10 @@ task: ulimit -c unlimited meson test $MTEST_ARGS --quiet --suite setup export LD_LIBRARY_PATH="$(pwd)/build/tmp_install/usr/local/pgsql/lib/:$LD_LIBRARY_PATH" - mkdir -p build/testrun + mkdir -p build/testrun ${PG_FAILED_TESTDIR} build/tmp_install/usr/local/pgsql/bin/initdb -N build/runningcheck --no-instructions -A trust echo "include '$(pwd)/src/tools/ci/pg_ci_base.conf'" >> build/runningcheck/postgresql.conf - build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start + build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l ${PG_FAILED_TESTDIR}/runningcheck.log start meson test $MTEST_ARGS --num-processes ${TEST_JOBS} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop EOF @@ -390,7 +391,7 @@ task: test_world_32_script: | su postgres <<-EOF ulimit -c unlimited - PYTHONCOERCECLOCALE=0 LANG=C meson test $MTEST_ARGS -C build-32 --num-processes ${TEST_JOBS} + PYTHONCOERCECLOCALE=0 LANG=C PG_FAILED_TESTDIR=`pwd`/failed.build-32 meson test $MTEST_ARGS -C build-32 --num-processes ${TEST_JOBS} EOF on_failure: diff --git a/src/tools/testwrap b/src/tools/testwrap index 7a64fe76a2d..445ac595afc 100755 --- a/src/tools/testwrap +++ b/src/tools/testwrap @@ -44,4 +44,10 @@ if sp.returncode == 0: else: print('# test failed') open(os.path.join(testdir, 'test.fail'), 'x') + faileddir = os.getenv('PG_FAILED_TESTDIR') + if faileddir: + parentdir = os.path.dirname(testdir) + newdest = os.path.join(faileddir, os.path.basename(parentdir), os.path.basename(testdir)) + shutil.copytree(testdir, newdest) + sys.exit(sp.returncode) -- 2.25.1 [text/x-diff] 0006-cirrus-ccache-use-G-rather-than-GB-suffix.patch (739B, ../../[email protected]/7-0006-cirrus-ccache-use-G-rather-than-GB-suffix.patch) download | inline diff: From e4534821ef5c1997ff34a2515884c5592f171e0b Mon Sep 17 00:00:00 2001 From: Justin Pryzby <[email protected]> Date: Sun, 13 Nov 2022 14:03:27 -0600 Subject: [PATCH 6/9] cirrus/ccache: use G rather than GB suffix.. the former being the documented spelling. --- .cirrus.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.cirrus.yml b/.cirrus.yml index fd5585e5818..a9fa4b5af27 100644 --- a/.cirrus.yml +++ b/.cirrus.yml @@ -667,7 +667,7 @@ task: # Use larger ccache cache, as this task compiles with multiple compilers / # flag combinations - CCACHE_MAXSIZE: "1GB" + CCACHE_MAXSIZE: "1G" CCACHE_DIR: "/tmp/ccache_dir" LINUX_CONFIGURE_FEATURES: *LINUX_CONFIGURE_FEATURES -- 2.25.1 [text/x-diff] 0007-cirrus-code-coverage.patch (4.5K, ../../[email protected]/8-0007-cirrus-code-coverage.patch) download | inline diff: From 185d1c3ed13f5472fea32fd15f55158721b7245b Mon Sep 17 00:00:00 2001 From: Justin Pryzby <[email protected]> Date: Mon, 17 Jan 2022 00:54:28 -0600 Subject: [PATCH 7/9] cirrus: code coverage Some alternatives: - could build with "--coverage -fprofile-filter-files=", but that means that ccache will never work (both because it doesn't support that option and also because the arguments will be different for every patch). - could use ninja coverage-html, but that can't filter only changed files, and would take a long time to upload a lot of useless files. https://www.postgresql.org/message-id/202202111821.w3gqblvfp4pr%40alvherre.pgsql https://www.postgresql.org/message-id/flat/[email protected] ci-os-only: freebsd --- .cirrus.yml | 18 +++++++++++++++- src/tools/ci/code-coverage-report | 35 +++++++++++++++++++++++++++++++ 2 files changed, 52 insertions(+), 1 deletion(-) create mode 100755 src/tools/ci/code-coverage-report diff --git a/.cirrus.yml b/.cirrus.yml index a9fa4b5af27..ec15263a665 100644 --- a/.cirrus.yml +++ b/.cirrus.yml @@ -28,6 +28,14 @@ env: TEMP_CONFIG: ${CIRRUS_WORKING_DIR}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl + # The commit that this branch is rebased on. There's no easy way to find this. + # This does the right thing for cfbot, which always squishes all patches into a single commit. + # And does the right thing for any 1-patch commits. + # Patch series manually submitted to cirrus would benefit from setting this to the + # number of patches in the series (or directly to the commit the series was rebased on). + #BASE_COMMIT: HEAD~1 + # For demo purposes: + BASE_COMMIT: HEAD~11 # What files to preserve in case tests fail on_failure_ac: &on_failure_ac @@ -157,6 +165,7 @@ task: uname -a ulimit -a -H && ulimit -a -S export + git diff --name-only "$BASE_COMMIT" ccache_cache: folder: $CCACHE_DIR @@ -170,7 +179,7 @@ task: chown root:postgres /tmp/cores sysctl kern.corefile='/tmp/cores/%N.%P.core' setup_additional_packages_script: | - #pkg install -y ... + pkg install -y lcov # NB: Intentionally build without -Dllvm. The freebsd image size is already # large enough to make VM startup slow @@ -178,6 +187,7 @@ task: su postgres <<-EOF meson setup \ --buildtype=debug \ + -Db_coverage=true \ -Dcassert=true -Dssl=openssl -Duuid=bsd -Dtcl_version=tcl86 -Ddtrace=auto \ -DPG_TEST_EXTRA="$PG_TEST_EXTRA" \ -Dextra_lib_dirs=/usr/local/lib -Dextra_include_dirs=/usr/local/include/ \ @@ -188,10 +198,16 @@ task: test_world_script: | su postgres <<-EOF + set -e ulimit -c unlimited meson test $MTEST_ARGS --num-processes ${TEST_JOBS} + # Create coverage report for files changed since the base commit. + time ./src/tools/ci/code-coverage-report "$BASE_COMMIT" ./build ./coverage EOF + coverage_artifacts: + path: 'coverage/**' + # test runningcheck, freebsd chosen because it's currently fast enough test_running_script: | su postgres <<-EOF diff --git a/src/tools/ci/code-coverage-report b/src/tools/ci/code-coverage-report new file mode 100755 index 00000000000..4f29d3f17a7 --- /dev/null +++ b/src/tools/ci/code-coverage-report @@ -0,0 +1,35 @@ +#! /bin/sh +# Called during CI to generate a code coverage report of changed files. +set -e + +base_branch=$1 +build_dir=$2 +outdir=$3 + +changed=`git diff --name-only "$base_branch" '*.c'` +[ -z "$changed" ] && exit 0 # Nothing changed + +mkdir "$outdir" + +# Coverage is shown only for changed files +# This is useful to see coverage of newly-added code, but won't +# show added/lost coverage in files which this patch doesn't modify. + +# This could be used to map from object file back to source file: +# readelf --debug-dump=info src/backend/postgres_lib.a.p/utils_adt_array_userfuncs.c.o |awk '/DW_AT_name/{print $NF;exit}' +# gcov ./src/port/libpgport_shlib.a.p/inet_net_ntop.c.gcno --stdout + +gcov=$outdir/coverage.gcov +lcov --quiet --capture --directory "$build_dir" >"$gcov.all" + +# Filter to include only changed files +echo "$changed" |sed 's,^,*/,' | + xargs -rt lcov --extract "$gcov.all" >"$gcov" + +ls -l "$outdir" + +# Exit successfully if no relevant files were changed +[ -s "$gcov" ] || exit 0 + +genhtml "$gcov" --show-details --legend --quiet --num-spaces=4 --output-directory "$outdir" --title="Coverage report of files changed since: $base_branch" +cp "$outdir"/index.html "$outdir"/00-index.html -- 2.25.1 [text/x-diff] 0008-cirrus-upload-changed-html-docs-as-artifacts.patch (2.7K, ../../[email protected]/9-0008-cirrus-upload-changed-html-docs-as-artifacts.patch) download | inline diff: From 5dace84a03882f7c0a274102fe497bfe9f606a76 Mon Sep 17 00:00:00 2001 From: Justin Pryzby <[email protected]> Date: Sat, 26 Feb 2022 19:39:10 -0600 Subject: [PATCH 8/9] cirrus: upload changed html docs as artifacts This could be done on the client side (cfbot). One advantage of doing it here is that fewer docs are uploaded - many patches won't upload docs at all. https://www.postgresql.org/message-id/flat/[email protected] https://www.postgresql.org/message-id/CAB8KJ=i4qmEuopQ+PCSMBzGd4O-Xv0FCnC+q1x7hN9hsdvkBug@mail.gmail.com https://cirrus-ci.com/task/5396696388599808 ci-os-only: html --- .cirrus.yml | 16 +++++++++++++++- src/tools/ci/copy-changed-docs | 29 +++++++++++++++++++++++++++++ 2 files changed, 44 insertions(+), 1 deletion(-) create mode 100755 src/tools/ci/copy-changed-docs diff --git a/.cirrus.yml b/.cirrus.yml index ec15263a665..57359f43867 100644 --- a/.cirrus.yml +++ b/.cirrus.yml @@ -701,6 +701,7 @@ task: gcc -v clang -v export + git diff --name-only "$BASE_COMMIT" ccache_cache: folder: $CCACHE_DIR @@ -775,7 +776,7 @@ task: time make -s -j${BUILD_JOBS} world-bin ### - # Verify docs can be built + # Verify docs can be built, and upload changed docs as artifacts ### # XXX: Only do this if there have been changes in doc/ since last build always: @@ -787,6 +788,19 @@ task: CLANG="ccache clang" make -s -j${BUILD_JOBS} clean time make -s -j${BUILD_JOBS} -C doc + cp -r doc new-docs + + # Re-build HTML docs from the base commit. + git checkout "$BASE_COMMIT" -- doc + make -s -C doc clean + time make -s -C doc html + cp -r doc old-docs + + copy_changed_docs_script: + - src/tools/ci/copy-changed-docs "old-docs" "new-docs" "html_docs" + + html_docs_artifacts: + paths: ['html_docs/*.html', 'html_docs/*.png', 'html_docs/*.css'] ### # Verify headerscheck / cpluspluscheck succeed diff --git a/src/tools/ci/copy-changed-docs b/src/tools/ci/copy-changed-docs new file mode 100755 index 00000000000..1c921a8df6f --- /dev/null +++ b/src/tools/ci/copy-changed-docs @@ -0,0 +1,29 @@ +#! /bin/sh +# Copy HTML which differ between $old and $new into $outdir +set -e + +old=$1 +new=$2 +outdir=$3 + +# The index is large and changes often +skippages="bookindex.html" + +mkdir "$outdir" +cp "$new"/src/sgml/html/*.css "$new"/src/sgml/html/*.svg "$outdir" + +changed=`git diff --no-index --name-only "$old"/src/sgml/html "$new"/src/sgml/html` || + [ $? -eq 1 ] + +for f in $changed +do + # Avoid removed files + [ -f "$f" ] || continue + + echo "$f" |grep -Ew "$skippages" >/dev/null && + continue + + cp -v "$f" "$outdir" +done + +exit 0 -- 2.25.1 [text/x-diff] 0009-html-index-file.patch (3.2K, ../../[email protected]/10-0009-html-index-file.patch) download | inline diff: From 852360330ef7e41edfb4d591ef246435aa956731 Mon Sep 17 00:00:00 2001 From: Justin Pryzby <[email protected]> Date: Mon, 28 Feb 2022 23:18:19 -0600 Subject: [PATCH 9/9] +html index file This allows linking to the artifacts from the last successful build. //freebsd ci-os-only: warnings --- .cirrus.yml | 4 ++-- src/tools/ci/copy-changed-docs | 42 ++++++++++++++++++++++++++++++++-- 2 files changed, 42 insertions(+), 4 deletions(-) diff --git a/.cirrus.yml b/.cirrus.yml index 57359f43867..f36d7d110e0 100644 --- a/.cirrus.yml +++ b/.cirrus.yml @@ -674,8 +674,8 @@ task: # succeeds. This is particularly important for this task as we intentionally # use always: to continue after failures. Task that did not run count as a # success, so we need to recheck SanityChecks's condition here ... - depends_on: SanityCheck - only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' + #depends_on: SanityCheck + only_if: $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:.*warnings' env: CPUS: 4 diff --git a/src/tools/ci/copy-changed-docs b/src/tools/ci/copy-changed-docs index 1c921a8df6f..57dd2823a50 100755 --- a/src/tools/ci/copy-changed-docs +++ b/src/tools/ci/copy-changed-docs @@ -5,6 +5,10 @@ set -e old=$1 new=$2 outdir=$3 +branch=$CIRRUS_BRANCH + +# The index is large and changes often +skippages="bookindex.html" # The index is large and changes often skippages="bookindex.html" @@ -12,6 +16,19 @@ skippages="bookindex.html" mkdir "$outdir" cp "$new"/src/sgml/html/*.css "$new"/src/sgml/html/*.svg "$outdir" +# The index is useful to allow a static link (not specific to a cirrus run) to the artifacts for the most-recent, successful CI run for a branch +branchurl=https://api.cirrus-ci.com/v1/artifact/github/$CIRRUS_REPO_FULL_NAME/$CIRRUS_TASK_NAME/html_docs/html_docs/00-index.html?branch=$branch + +index="$outdir/00-index.html" +cat >"$index" <<EOF +<html> +<head><title>Index of docs changed since: $BASE_COMMIT</title></head> +<body> +<!-- A link to documentation for the most recent successful build: $branchurl --> +<h1>Index of docs changed since: $BASE_COMMIT</h1> +<ul> +EOF + changed=`git diff --no-index --name-only "$old"/src/sgml/html "$new"/src/sgml/html` || [ $? -eq 1 ] @@ -23,7 +40,28 @@ do echo "$f" |grep -Ew "$skippages" >/dev/null && continue - cp -v "$f" "$outdir" -done + cp -v "$f" "$outdir" >&2 + fn=${f##*/} + # ?branch=... is needed when accessing the artifacts for the static link for the branch + # It's not needed and ignored if accessing artifacts for *this* CI run + echo "<li><a href='$fn?branch=$branch'>$fn</a>" +done >>"$index" + +github=https://github.com/$CIRRUS_REPO_FULL_NAME/commit +cirrus=https://cirrus-ci.com/build + +cat >>"$index" <<EOF +</ul> +<hr> +<code> +<br>This file was written on: `date --rfc-822 --utc` +<br>CIRRUS_CHANGE_TITLE: $CIRRUS_CHANGE_TITLE +<br>CIRRUS_CHANGE_IN_REPO: <a href="$github/$CIRRUS_CHANGE_IN_REPO">$CIRRUS_CHANGE_IN_REPO</a> +<br>CIRRUS_BUILD_ID: <a href="$cirrus/$CIRRUS_BUILD_ID">$CIRRUS_BUILD_ID</a> +<br>CIRRUS_LAST_GREEN_CHANGE: <a href="$github/$CIRRUS_LAST_GREEN_CHANGE">$CIRRUS_LAST_GREEN_CHANGE</a> +<br>CIRRUS_LAST_GREEN_BUILD_ID: <a href="$cirrus/$CIRRUS_LAST_GREEN_BUILD_ID">$CIRRUS_LAST_GREEN_BUILD_ID</a> +</code> +</body></html> +EOF exit 0 -- 2.25.1 ^ permalink raw reply [nested|flat] 3+ messages in thread
* Re: CI and test improvements @ 2023-03-13 06:39 Peter Eisentraut <[email protected]> parent: Justin Pryzby <[email protected]> 0 siblings, 0 replies; 3+ messages in thread From: Peter Eisentraut @ 2023-03-13 06:39 UTC (permalink / raw) To: Justin Pryzby <[email protected]>; Thomas Munro <[email protected]>; +Cc: Tom Lane <[email protected]>; Andres Freund <[email protected]>; Andrew Dunstan <[email protected]>; [email protected]; Noah Misch <[email protected]>; Michael Paquier <[email protected]>; Anastasia Lubennikova <[email protected]>; Robert Haas <[email protected]>; Melanie Plageman <[email protected]>; Daniel Gustafsson <[email protected]>; samay sharma <[email protected]> On 03.02.23 15:26, Justin Pryzby wrote: > rebased, and re-including a patch to show code coverage of changed > files. This constant flow of patches under one subject doesn't lend itself well to the commit fest model of trying to finish things up. I can't quite tell which of these patches are ready and agreed upon, and which ones are work in progress or experimental. > e4534821ef5 cirrus/ccache: use G rather than GB suffix.. This one seems obvious. I have committed it. > 9baf41674ad pg_upgrade: tap test: exercise --link and --clone This seems like a good idea. > 7e09035f588 WIP: ci/meson: allow showing only failed tests .. I'm not sure I like this one. I sometimes look up the logs of non-failed tests to compare them with failed tests, to get context to could lead to failures. Maybe we can make this behavior adjustable. But I've not been bothered by the current behavior. ^ permalink raw reply [nested|flat] 3+ messages in thread
end of thread, other threads:[~2023-03-13 06:39 UTC | newest] Thread overview: 3+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2020-09-29 14:19 [PATCH v45 7/7] Exclude pg_stat directory from base backup Kyotaro Horiguchi <[email protected]> 2023-02-03 14:26 Re: CI and test improvements Justin Pryzby <[email protected]> 2023-03-13 06:39 ` Re: CI and test improvements Peter Eisentraut <[email protected]>
This inbox is served by agora; see mirroring instructions for how to clone and mirror all data and code used for this inbox