agora inbox for [email protected]help / color / mirror / Atom feed
[PATCH 2/4] Use double quotes in message 205+ messages / 2 participants [nested] [flat]
* [PATCH 2/4] Use double quotes in message @ 2026-05-26 07:17 Kyotaro Horiguchi <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Kyotaro Horiguchi @ 2026-05-26 07:17 UTC (permalink / raw) Replace backquotes around a symbol name in a user-facing message with double quotes for consistency with other messages. --- src/interfaces/libpq/fe-protocol3.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/interfaces/libpq/fe-protocol3.c b/src/interfaces/libpq/fe-protocol3.c index 840e018cd18..4efa8a16ab1 100644 --- a/src/interfaces/libpq/fe-protocol3.c +++ b/src/interfaces/libpq/fe-protocol3.c @@ -1560,7 +1560,7 @@ pqGetNegotiateProtocolVersion3(PGconn *conn) */ if (expect_test_protocol_negotiation && !found_test_protocol_negotiation) { - libpq_append_conn_error(conn, "server did not report the unsupported `_pq_.test_protocol_negotiation` parameter in its protocol negotiation message"); + libpq_append_conn_error(conn, "server did not report the unsupported \"_pq_.test_protocol_negotiation\" parameter in its protocol negotiation message"); goto failure; } -- 2.47.3 ----Next_Part(Thu_May_28_12_16_22_2026_214)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0003-Add-missing-period-to-HINT-message.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
* [PATCH v13a 7/9] ci: get build/test concurrency from environment @ 2026-06-10 18:49 Andres Freund <[email protected]> 0 siblings, 0 replies; 205+ messages in thread From: Andres Freund @ 2026-06-10 18:49 UTC (permalink / raw) Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0 --mr2uqtieh2e2xvha Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13a-0008-ci-Make-runs-on-overridable.patch" ^ permalink raw reply [nested|flat] 205+ messages in thread
end of thread, other threads:[~2026-06-10 18:49 UTC | newest] Thread overview: 205+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2026-05-26 07:17 [PATCH 2/4] Use double quotes in message Kyotaro Horiguchi <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]> 2026-06-10 18:49 [PATCH v13a 7/9] ci: get build/test concurrency from environment Andres Freund <[email protected]>
This inbox is served by agora; see mirroring instructions for how to clone and mirror all data and code used for this inbox