agora inbox for pgsql-hackers@postgresql.orghelp / color / mirror / Atom feed
[PATCH v3 3/3] Better express platform requirements in s_lock.h. 249+ messages / 2 participants [nested] [flat]
* [PATCH v3 3/3] Better express platform requirements in s_lock.h. @ 2026-05-07 20:32 Nathan Bossart <nathan@postgresql.org> 0 siblings, 0 replies; 249+ messages in thread From: Nathan Bossart @ 2026-05-07 20:32 UTC (permalink / raw) --- src/include/storage/s_lock.h | 56 ++++++++++++++++++++---------------- 1 file changed, 32 insertions(+), 24 deletions(-) diff --git a/src/include/storage/s_lock.h b/src/include/storage/s_lock.h index fb872edd2f0..c4369ee4ff6 100644 --- a/src/include/storage/s_lock.h +++ b/src/include/storage/s_lock.h @@ -44,23 +44,16 @@ * atomic test-and-set only when it appears free. * * TAS() and TAS_SPIN() are NOT part of the API, and should never be called - * directly. - * - * CAUTION: on some platforms TAS() and/or TAS_SPIN() may sometimes report - * failure to acquire a lock even when the lock is not locked. For example, - * on Alpha TAS() will "fail" if interrupted. Therefore a retry loop must - * always be used, even if you are certain the lock is free. + * directly. If a platform-specific TAS() is defined, the platform must + * _not_ define its own S_LOCK(). Conversely, if a platform-specific + * S_LOCK() is defined, the platform must _not_ define its own TAS(), but + * it does need to define its own TAS_SPIN(). Currently, all supported + * platforms define TAS() and use the default S_LOCK() implementation, so + * that is probably a good place to start if adding a new one. * * It is the responsibility of these macros to make sure that the compiler * does not re-order accesses to shared memory to precede the actual lock - * acquisition, or follow the lock release. Prior to PostgreSQL 9.5, this - * was the caller's responsibility, which meant that callers had to use - * volatile-qualified pointers to refer to both the spinlock itself and the - * shared data being accessed within the spinlocked critical section. This - * was notationally awkward, easy to forget (and thus error-prone), and - * prevented some useful compiler optimizations. For these reasons, we - * now require that the macros themselves prevent compiler re-ordering, - * so that the caller doesn't need to take special precautions. + * acquisition, or follow the lock release. * * On platforms with weak memory ordering, the TAS(), TAS_SPIN(), and * S_UNLOCK() macros must further include hardware-level memory fence @@ -72,7 +65,7 @@ * * On most supported platforms, TAS() uses a tas() function written * in assembly language to execute a hardware atomic-test-and-set - * instruction. Equivalent OS-supplied mutex routines could be used too. + * instruction. Equivalent compiler intrinsics are another popular option. * * * Portions Copyright (c) 1996-2026, PostgreSQL Global Development Group @@ -642,20 +635,25 @@ spin_delay(void) #endif /* !defined(TAS) */ -/* Blow up if we didn't have any way to do spinlocks */ -#ifndef TAS -#error PostgreSQL does not have spinlock support on this platform. Please report this to pgsql-bugs@lists.postgresql.org. -#endif - - /* * Default Definitions - override these above as needed. */ -#if !defined(S_LOCK) +/* + * Make sure S_LOCK is defined, either explicitly for the platform or via a TAS + * macro for the platform. Exactly one of either S_LOCK or TAS should be + * defined for a supported platform at this point in the file. + */ +#if defined(S_LOCK) +#if defined(TAS) +#error Both TAS and S_LOCK defined on this platform. Please report this to pgsql-bugs@lists.postgresql.org. +#endif +#elif defined(TAS) #define S_LOCK(lock) \ (TAS(lock) ? s_lock((lock), __FILE__, __LINE__, __func__) : 0) -#endif /* S_LOCK */ +#else +#error Neither TAS nor S_LOCK defined on this platform. Please report this to pgsql-bugs@lists.postgresql.org. +#endif #if !defined(S_UNLOCK) /* @@ -687,9 +685,19 @@ extern void s_unlock(volatile slock_t *lock); #define SPIN_DELAY() ((void) 0) #endif /* SPIN_DELAY */ +/* + * We can only define TAS_SPIN if TAS was defined. Otherwise, the platform + * defined its own S_LOCK without TAS, and therefore is responsible for + * defining its own TAS_SPIN as well. (Note that we currently do not have any + * platforms that don't define TAS.) + */ #if !defined(TAS_SPIN) +#if defined(TAS) #define TAS_SPIN(lock) TAS(lock) -#endif /* TAS_SPIN */ +#else +#error Neither TAS nor TAS_SPIN defined on this platform. Please report this to pgsql-bugs@lists.postgresql.org. +#endif /* TAS */ +#endif /* ! TAS_SPIN */ /* -- 2.50.1 (Apple Git-155) --IHFkEDr8CkxIje3R-- ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
end of thread, other threads:[~2026-06-10 18:49 UTC | newest] Thread overview: 249+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2026-05-07 20:32 [PATCH v3 3/3] Better express platform requirements in s_lock.h. Nathan Bossart <nathan@postgresql.org> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <andres@anarazel.de>
This inbox is served by agora; see mirroring instructions for how to clone and mirror all data and code used for this inbox