agora inbox for [email protected]
help / color / mirror / Atom feedFrom: Andres Freund <[email protected]>
Subject: [PATCH v13a 7/9] ci: get build/test concurrency from environment
Date: Wed, 10 Jun 2026 14:49:28 -0400
Previously we hardcoded the amount of concurrency in the CI definition. This
practice IIRC originated from a) using make, where the concurrency is not
sourced from the current system b) early cirrus-ci macos runners, which slowed
down a lot with full concurrency.
Hardcoding the concurrency is problematic as e.g. private repositories have
lower concurrency and larger github runners have more cores.
For most of the jobs we can rely on just using meson's and ninja's logic for
getting the current core count. For the autoconf we can use the nproc helper.
To make this at least somewhat understandable, I added the existing sysinfo
step to show the number of cores (and made it a bit easier to understand the
information already printed).
Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33
---
.github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++--------------
1 file changed, 29 insertions(+), 18 deletions(-)
diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml
index ab7950f6f14..9384ec0d902 100644
--- a/.github/workflows/pg-ci.yml
+++ b/.github/workflows/pg-ci.yml
@@ -40,14 +40,6 @@ env:
# concurrent jobs and retrying older runs have a chance of working.
CLONE_DEPTH: 500
- # At the moment all jobs use 4vcore runners, and none seems to benefit from
- # increasing concurrency further.
- BUILD_JOBS: 4
-
- # It's possible that some jobs benefit from an increased test concurrency,
- # but a default of 4 is a safe bet. Individual jobs can override.
- TEST_JOBS: 4
-
CCACHE_MAXSIZE: "250M"
CCACHE_DIR: ${{ github.workspace }}/ccache_dir
@@ -215,10 +207,19 @@ jobs:
- &nix_sysinfo_step
name: sysinfo
run: |
- id
- uname -a
+ echo -n "Number of cores: "
+ nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu
+
+ echo "id: $(id)"
+ echo "uname: $(uname -a)"
+
+ echo ::group::ulimit
ulimit -a -H && ulimit -a -S
+ echo ::endgroup::
+
+ echo ::group::env
env
+ echo ::endgroup::
- name: Parse ci-os-only
id: os
@@ -352,7 +353,7 @@ jobs:
- name: Build
shell: *su_postgres_shell
run: &ninja_build_cmd |
- ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}}
+ ninja -C build ${{env.MBUILD_TARGET}}
ninja -C build -t missingdeps
# Decide if it's worth uploading a new version of the ccache cache. If
@@ -405,7 +406,7 @@ jobs:
meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1
echo ::endgroup::
- meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}}
+ meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}}
- &linux_collect_cores_step
name: Core backtraces
@@ -511,7 +512,7 @@ jobs:
- name: Build
shell: *su_postgres_shell
run: |
- make -s -j${BUILD_JOBS} world-bin
+ make -s -j$(nproc) world-bin
- *ccache_decide_save_step
- *ccache_save_step
@@ -519,7 +520,7 @@ jobs:
- name: Test world
shell: *su_postgres_shell
run: |
- make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS}
+ make -s ${CHECK} ${CHECKFLAGS} -j$(nproc)
- *linux_collect_cores_step
- *upload_logs_step
@@ -611,7 +612,7 @@ jobs:
build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start
# Run the tests supporting running against an already running
- meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running
+ meson test ${{env.MTEST_ARGS}} --setup running
build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop
@@ -893,9 +894,19 @@ jobs:
- name: Sysinfo
run: |
+ echo Number of cores:
+ bash -c nproc
+
+ echo codepage:
chcp
+
+ echo ::group::systeminfo
systeminfo
+ echo ::endgroup::
+
+ echo ::group::env
set
+ echo ::endgroup::
# The TAP tests build an initdb template under build/tmp_install and
# then `robocopy` it into per-test data directories. Robocopy with the
@@ -1198,7 +1209,7 @@ jobs:
meson setup -Dwerror=true $CONF build-meson
echo "::endgroup::"
- ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}}
+ ninja -C build-meson --quiet ${{env.MBUILD_TARGET}}
# gcc, cassert on, dtrace off
- name: gcc warnings + (cassert)
@@ -1266,8 +1277,8 @@ jobs:
CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang"
echo "::endgroup::"
- make -s -j${{env.BUILD_JOBS}} clean
- make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \
+ make -s -j$(nproc) clean
+ make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \
headerscheck cpluspluscheck \
EXTRAFLAGS='-fmax-errors=10'
--
2.54.0.450.g9ac3f193c0
--mr2uqtieh2e2xvha
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v13a-0008-ci-Make-runs-on-overridable.patch"
view thread (102+ messages) latest in thread
reply
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: [email protected]
Cc: [email protected]
Subject: Re: [PATCH v13a 7/9] ci: get build/test concurrency from environment
In-Reply-To: <no-message-id-873667@localhost>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox