pg.ddx.io pgsql-hackers@postgresql.org mailing list archive
help / color / mirror / Atom feedCI slowdown due to PG_TEST_INITDB_EXTRA_OPTS
14+ messages / 7 participants
[nested] [flat]
* CI slowdown due to PG_TEST_INITDB_EXTRA_OPTS
@ 2026-07-18 16:38 Andres Freund <andres@anarazel.de>
0 siblings, 1 reply; 14+ messages in thread
From: Andres Freund @ 2026-07-18 16:38 UTC (permalink / raw)
To: pgsql-hackers; +Cc: Peter Eisentraut <peter.eisentraut@enterprisedb.com>; Nazir Bilal Yavuz <byavuz81@gmail.com>
Hi,
I was once more wondering why CI is so slow. While thinking about what
measurements to take, I think I figured out a major source:
Many of the jobs use PG_TEST_INITDB_EXTRA_OPTS. Unfortunately, using that
turns out to disable the initdb template mechanism:
sub init
{
...
if (defined $initdb_extra_opts_env)
{
push @{ $params{extra} }, shellwords($initdb_extra_opts_env);
}
...
# If available, if there aren't any parameters and if force_initdb is
# disabled, use a previously initdb'd cluster as a template by copying it.
# For a lot of tests, that's substantially cheaper. It does not seem
# worth figuring out whether extra parameters affect compatibility, so
# initdb is forced if any are defined.
#
# There's very similar code in pg_regress.c, but we can't easily
# deduplicate it until we require perl at build time.
if ( $params{force_initdb}
or defined $params{extra}
or !defined $ENV{INITDB_TEMPLATE})
{
note("initializing database system by running initdb");
To see the impact of that, I just hacked up a commit that comments out all the
use of PG_TEST_INITDB_EXTRA_OPTS. That's of course not a fair comparison,
several of the options passed to PG_TEST_INITDB_EXTRA_OPTS do increase
overhead, but it's an approximation.
master: https://github.com/anarazel/postgres/actions/runs/29593721971
hack: https://github.com/anarazel/postgres/actions/runs/29597834725
(note that for the hack the compiles are cached, so you can't look at the
overall runtimes)
master hack
autoconf 14m52s 8m45s
meson-32 8m13s 6m52s
meson-64 19m19s 13m27s
macos 19m24s 9m07s
mingw 15m41s 19m05s
vs-slice-1 15m59s 14m19s
vs-slice-2 10m39s 13m55s
Note that mingw and vs did not use PG_TEST_INITDB_EXTRA_OPTS, so their test
timing is unrelated. Which also shows that the rest of the numbers are not
entirely to be trusted.
But I can also reproduce decent improvements locally, so I'm pretty confident
not using PG_TEST_INITDB_EXTRA_OPTS will yield significant improvements:
To make the comparison fairer, I'm comparing not using
PG_TEST_INITDB_EXTRA_OPTS
with using
PG_TEST_INITDB_EXTRA_OPTS='-c debug_copy_parse_plan_trees=off'
which shouldn't change any runtime behaviour.
PG_TEST_INITDB_EXTRA_OPTS='-c debug_copy_parse_plan_trees=off':
1794.54user 366.10system 2:11.37elapsed 1644%CPU (0avgtext+0avgdata 153580maxresident)k
52160inputs+69971904outputs (612major+26216385minor)pagefaults 0swaps
without:
435.36user 293.50system 1:48.81elapsed 669%CPU (0avgtext+0avgdata 153508maxresident)k
429472inputs+34543792outputs (780major+20764791minor)pagefaults 0swaps
So that's a ~3x increase in CPU time, no wonder that slows down the tests.
And that's without even using sanitizers, which make initdb a good bit slower!
Now, the slightly harder question is how to do this without just ripping out
the PG_TEST_INITDB_EXTRA_OPTS. Looks like we started using that for CI with:
commit a292c98d62ddc0ad681f772ab91bf68ee399cb4b
Author: Peter Eisentraut <peter@eisentraut.org>
Date: 2024-08-01 09:37:44 +0200
Convert node test compile-time settings into run-time parameters
I guess the reason Peter didn't use TEMP_CONFIG is that it's kinda annoying to
do properly, due TEMP_CONFIG already being set to src/tools/ci/pg_ci_base.conf.
Without adding new pg_regress.c and Cluster.pm code, it seems we would have to
create a conf file that includes src/tools/ci/pg_ci_base.conf and also adds
the other options. But that's somewhat annoying to do :(
We could add an include_if_exists to src/tools/ci/pg_ci_base.conf, but I'm not
sure what absolute path would be portable enough, and I don't think we allow
environment variables or such.
Any better ideas?
On macos we probably also could get a decent further boost by using cp -c when
copying the template data directory (which uses copy on write). On linux the
default is --reflink=auto, so it'd get used automatically, but CI doesn't run
on a CoW/reflink capable filesystem (locally I see pretty decent gains from
it) . On an m4 mac mini, I see about a 10% gain, but it's quite plausible
this would be larger in CI. But that's a separate thread, I guess.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 14+ messages in thread
* Re: CI slowdown due to PG_TEST_INITDB_EXTRA_OPTS
@ 2026-07-18 18:47 Tom Lane <tgl@sss.pgh.pa.us>
parent: Andres Freund <andres@anarazel.de>
0 siblings, 1 reply; 14+ messages in thread
From: Tom Lane @ 2026-07-18 18:47 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: pgsql-hackers@postgresql.org, Peter Eisentraut <peter.eisentraut@enterprisedb.com>; Nazir Bilal Yavuz <byavuz81@gmail.com>
Andres Freund <andres@anarazel.de> writes:
> I was once more wondering why CI is so slow. While thinking about what
> measurements to take, I think I figured out a major source:
> Many of the jobs use PG_TEST_INITDB_EXTRA_OPTS. Unfortunately, using that
> turns out to disable the initdb template mechanism:
> ...
> Any better ideas?
It looks like any one job mostly uses the same value of
PG_TEST_INITDB_EXTRA_OPTS. Could we record what was used to set up
the template, and allow using it if that matches?
regards, tom lane
^ permalink raw reply [nested|flat] 14+ messages in thread
* Re: CI slowdown due to PG_TEST_INITDB_EXTRA_OPTS
@ 2026-07-20 21:47 Jelte Fennema-Nio <postgres@jeltef.nl>
parent: Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 1 reply; 14+ messages in thread
From: Jelte Fennema-Nio @ 2026-07-20 21:47 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Andres Freund <andres@anarazel.de>; pgsql-hackers; Peter Eisentraut <peter.eisentraut@enterprisedb.com>; Nazir Bilal Yavuz <byavuz81@gmail.com>
On Sat, 18 Jul 2026 at 20:48, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> It looks like any one job mostly uses the same value of
> PG_TEST_INITDB_EXTRA_OPTS. Could we record what was used to set up
> the template, and allow using it if that matches?
Yeah, that seems to work pretty well. See attached.
I'm getting similar test timings in CI with this as Andres got for the
Linux runs that he did when removing PG_TEST_INITDB_EXTRA_OPTS.
For MacOS I see basically no improvement though, I guess that's probably
because the debug_xxx options actually incur significant overhead. Maybe
it's worth trying to move them to one of the faster runs. But that seems
like a separate discussion.
Attachments:
[text/x-patch] v1-0001-Keep-using-the-initdb-template-when-PG_TEST_INITD.patch (9.6K, ../../DK3Q8SC52NHB.SGQ8FXBIDLCP@jeltef.nl/2-v1-0001-Keep-using-the-initdb-template-when-PG_TEST_INITD.patch)
download | inline diff:
From 47ae7fd5ed7ef5104123c083a400f44fd5a13906 Mon Sep 17 00:00:00 2001
From: Jelte Fennema-Nio <postgres@jeltef.nl>
Date: Mon, 20 Jul 2026 21:54:19 +0200
Subject: [PATCH v1] Keep using the initdb template when
PG_TEST_INITDB_EXTRA_OPTS is set
Setting PG_TEST_INITDB_EXTRA_OPTS used to disable the initdb template
mechanism entirely: both Cluster.pm` and `pg_regress.c fell back to
running a full initdb for every test cluster, since they couldn't know
whether the template was compatible with the requested options. Because
CI sets this variable in most jobs, nearly every test paid the full
initdb cost, which is a major contributor to slow CI runs (initdb is
roughly 3x the CPU of copying the template, and worse under sanitizers).
This fixes this by using `PG_TEST_INITDB_EXTRA_OPTS` too when creating
the initdb template and storing the options that were used for the
template in a '{}.extra-opts' file. That way the runners can compare the
options in this file to their PG_TEST_INITDB_EXTRA_OPTS value. If it's
the same they can use the template, if not they need to run initdb.
---
meson.build | 23 +++++++++++-
src/Makefile.global.in | 12 +++++-
src/test/perl/PostgreSQL/Test/Cluster.pm | 42 +++++++++++++++------
src/test/regress/pg_regress.c | 47 +++++++++++++++++++++++-
4 files changed, 108 insertions(+), 16 deletions(-)
diff --git a/meson.build b/meson.build
index f4cde249242..74bf7b9f714 100644
--- a/meson.build
+++ b/meson.build
@@ -3901,18 +3901,37 @@ endif
# Create (and remove old) initdb template directory. Tests use that, where
# possible, to make it cheaper to run tests.
#
+# The template is created with PG_TEST_INITDB_EXTRA_OPTS applied, and the
+# variable's value is recorded in an ".extra-opts" file next to the template
+# directory. That allows tests to use the template when the variable is set,
+# as long as its value still matches, instead of falling back to a full
+# initdb.
+#
# Use python to remove the old cached initdb, as we cannot rely on a working
# 'rm' binary on windows.
test('initdb_cache',
python,
args: [
'-c', '''
+import os
+import shlex
import shutil
import sys
import subprocess
-shutil.rmtree(sys.argv[1], ignore_errors=True)
-sp = subprocess.run(sys.argv[2:] + [sys.argv[1]])
+template_dir = sys.argv[1]
+extra_opts = os.environ.get('PG_TEST_INITDB_EXTRA_OPTS', '')
+
+shutil.rmtree(template_dir, ignore_errors=True)
+try:
+ os.remove(template_dir + '.extra-opts')
+except FileNotFoundError:
+ pass
+
+sp = subprocess.run(sys.argv[2:] + shlex.split(extra_opts) + [template_dir])
+if sp.returncode == 0:
+ with open(template_dir + '.extra-opts', 'w') as f:
+ f.write(extra_opts)
sys.exit(sp.returncode)
''',
test_initdb_template,
diff --git a/src/Makefile.global.in b/src/Makefile.global.in
index cef1ad7f87d..ea0b218717a 100644
--- a/src/Makefile.global.in
+++ b/src/Makefile.global.in
@@ -442,7 +442,17 @@ ifeq ($(MAKELEVEL),0)
$(MAKE) -C '$(top_builddir)' DESTDIR='$(abs_top_builddir)'/tmp_install install >'$(abs_top_builddir)'/tmp_install/log/install.log 2>&1
$(MAKE) -j1 $(if $(CHECKPREP_TOP),-C $(CHECKPREP_TOP),) checkprep >>'$(abs_top_builddir)'/tmp_install/log/install.log 2>&1
- $(with_temp_install) initdb --auth trust --no-sync --no-instructions --lc-messages=C --no-clean '$(abs_top_builddir)'/tmp_install/initdb-template >>'$(abs_top_builddir)'/tmp_install/log/initdb-template.log 2>&1
+# The initdb template is created with PG_TEST_INITDB_EXTRA_OPTS applied, and
+# the variable's value is recorded in an ".extra-opts" file next to the
+# template directory. That allows tests to use the template when the variable
+# is set, as long as its value still matches. Run initdb through an extra
+# shell so that any quoting in PG_TEST_INITDB_EXTRA_OPTS is interpreted the
+# same way as in pg_regress.c and Cluster.pm. The template path is passed as
+# a positional argument ($0) instead of being embedded in the command string,
+# so that only PG_TEST_INITDB_EXTRA_OPTS gets that second round of shell
+# interpretation.
+ $(with_temp_install) sh -c "initdb --auth trust --no-sync --no-instructions --lc-messages=C --no-clean $$PG_TEST_INITDB_EXTRA_OPTS \"\$$0\"" '$(abs_top_builddir)'/tmp_install/initdb-template >>'$(abs_top_builddir)'/tmp_install/log/initdb-template.log 2>&1
+ printf '%s' "$$PG_TEST_INITDB_EXTRA_OPTS" >'$(abs_top_builddir)'/tmp_install/initdb-template.extra-opts
endif
endif
endif
diff --git a/src/test/perl/PostgreSQL/Test/Cluster.pm b/src/test/perl/PostgreSQL/Test/Cluster.pm
index 3eae4cf6281..8b1c3d73c39 100644
--- a/src/test/perl/PostgreSQL/Test/Cluster.pm
+++ b/src/test/perl/PostgreSQL/Test/Cluster.pm
@@ -639,38 +639,58 @@ sub init
$params{has_archiving} = 0 unless defined $params{has_archiving};
my $initdb_extra_opts_env = $ENV{PG_TEST_INITDB_EXTRA_OPTS};
+
+ my @initdb_opts = @{ $params{extra} // [] };
if (defined $initdb_extra_opts_env)
{
- push @{ $params{extra} }, shellwords($initdb_extra_opts_env);
+ push @initdb_opts, shellwords($initdb_extra_opts_env);
}
# This should override user-supplied initdb options.
if ($params{no_data_checksums})
{
- push @{ $params{extra} }, '--no-data-checksums';
+ push @initdb_opts, '--no-data-checksums';
}
mkdir $self->backup_dir;
mkdir $self->archive_dir;
- # If available, if there aren't any parameters and if force_initdb is
- # disabled, use a previously initdb'd cluster as a template by copying it.
- # For a lot of tests, that's substantially cheaper. It does not seem
- # worth figuring out whether extra parameters affect compatibility, so
- # initdb is forced if any are defined.
+ # If available, if there aren't any parameters the template wasn't
+ # created with and if force_initdb is disabled, use a previously initdb'd
+ # cluster as a template by copying it. For a lot of tests, that's
+ # substantially cheaper.
+ #
+ # The value of PG_TEST_INITDB_EXTRA_OPTS used when creating the template
+ # is recorded in an ".extra-opts" file next to the template directory, so
+ # the template stays usable when that variable is set, as long as its
+ # value matches. It does not seem worth figuring out whether
+ # test-supplied extra parameters affect compatibility, so initdb is
+ # forced if any are defined.
#
# There's very similar code in pg_regress.c, but we can't easily
# deduplicate it until we require perl at build time.
- if ( $params{force_initdb}
- or defined $params{extra}
- or !defined $ENV{INITDB_TEMPLATE})
+ my $template_usable = 0;
+ if ( !$params{force_initdb}
+ and !defined $params{extra}
+ and !$params{no_data_checksums}
+ and defined $ENV{INITDB_TEMPLATE})
+ {
+ my $extra_opts_file = $ENV{INITDB_TEMPLATE} . '.extra-opts';
+ my $template_opts =
+ -e $extra_opts_file
+ ? PostgreSQL::Test::Utils::slurp_file($extra_opts_file)
+ : '';
+ $template_usable = $template_opts eq ($initdb_extra_opts_env // '');
+ }
+
+ if (!$template_usable)
{
note("initializing database system by running initdb");
PostgreSQL::Test::Utils::system_or_bail(
'initdb', '--no-sync',
'--pgdata' => $pgdata,
'--auth' => 'trust',
- @{ $params{extra} });
+ @initdb_opts);
}
else
{
diff --git a/src/test/regress/pg_regress.c b/src/test/regress/pg_regress.c
index 21d00f792d2..d7adc211cc0 100644
--- a/src/test/regress/pg_regress.c
+++ b/src/test/regress/pg_regress.c
@@ -2138,6 +2138,45 @@ help(void)
printf(_("%s home page: <%s>\n"), PACKAGE_NAME, PACKAGE_URL);
}
+/*
+ * Check whether the initdb template was created with the same
+ * PG_TEST_INITDB_EXTRA_OPTS that are in effect now. The value used during
+ * template creation is recorded in an ".extra-opts" file next to the
+ * template directory; a missing file means the template was created without
+ * extra options. Without this check we'd have to fall back to a full initdb
+ * whenever PG_TEST_INITDB_EXTRA_OPTS is set, which is a lot more expensive.
+ */
+static bool
+initdb_template_matches(const char *initdb_template_dir,
+ const char *initdb_extra_opts_env)
+{
+ char path[MAXPGPATH];
+ char recorded[4096];
+ FILE *f;
+
+ snprintf(path, sizeof(path), "%s.extra-opts", initdb_template_dir);
+
+ recorded[0] = '\0';
+ if ((f = fopen(path, "r")) != NULL)
+ {
+ /*
+ * A value too long for the buffer can never compare equal, so treat
+ * truncation as a mismatch instead of comparing just the prefix.
+ */
+ size_t nread = fread(recorded, 1, sizeof(recorded) - 1, f);
+ bool truncated = (nread == sizeof(recorded) - 1 && !feof(f));
+
+ recorded[nread] = '\0';
+ fclose(f);
+
+ if (truncated)
+ return false;
+ }
+
+ return strcmp(recorded,
+ initdb_extra_opts_env ? initdb_extra_opts_env : "") == 0;
+}
+
int
regression_main(int argc, char *argv[],
init_function ifunc,
@@ -2401,13 +2440,17 @@ regression_main(int argc, char *argv[],
* Create data directory.
*
* If available, use a previously initdb'd cluster as a template by
- * copying it. For a lot of tests, that's substantially cheaper.
+ * copying it. For a lot of tests, that's substantially cheaper. The
+ * template is only usable if it was created with the same
+ * PG_TEST_INITDB_EXTRA_OPTS that are in effect now, which
+ * initdb_template_matches() checks.
*
* There's very similar code in Cluster.pm, but we can't easily de
* duplicate it until we require perl at build time.
*/
initdb_template_dir = getenv("INITDB_TEMPLATE");
- if (initdb_template_dir == NULL || nolocale || debug || initdb_extra_opts_env)
+ if (initdb_template_dir == NULL || nolocale || debug ||
+ !initdb_template_matches(initdb_template_dir, initdb_extra_opts_env))
{
note("initializing database system by running initdb");
--
2.54.0
^ permalink raw reply [nested|flat] 14+ messages in thread
* Re: CI slowdown due to PG_TEST_INITDB_EXTRA_OPTS
@ 2026-07-21 00:56 Andres Freund <andres@anarazel.de>
parent: Jelte Fennema-Nio <postgres@jeltef.nl>
0 siblings, 2 replies; 14+ messages in thread
From: Andres Freund @ 2026-07-21 00:56 UTC (permalink / raw)
To: Jelte Fennema-Nio <postgres@jeltef.nl>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; pgsql-hackers@postgresql.org, Peter Eisentraut <peter.eisentraut@enterprisedb.com>; Nazir Bilal Yavuz <byavuz81@gmail.com>
Hi,
On 2026-07-20 23:47:16 +0200, Jelte Fennema-Nio wrote:
> On Sat, 18 Jul 2026 at 20:48, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> > It looks like any one job mostly uses the same value of
> > PG_TEST_INITDB_EXTRA_OPTS. Could we record what was used to set up
> > the template, and allow using it if that matches?
That seems like a somewhat complicated mechanism. But mainly I feel like it's
a workaround around the config-customization-via-env-var having to go through
exactly one file. None of the uses of PG_TEST_INITDB_EXTRA_OPTS in CI actually
needs it to be an initdb option, it's just a way to set configuration options
for the server via an environment variable.
I wonder if the best thing would be to add PG_TEST_SERVER_POSTGRES_OPTS or
such, which Cluster.pm and regress.c would add to postgresql.conf.
> For MacOS I see basically no improvement though, I guess that's probably
> because the debug_xxx options actually incur significant overhead. Maybe
> it's worth trying to move them to one of the faster runs. But that seems
> like a separate discussion.
Hm. I see very consistent improvements locally on an m4 mac mini. On CI the
runtime of the macos job is crazily variable, e.g. the last few runs on master
had these runtimes for macos' "Test world":
14m59s, 17m24s, 12m13s, 18m38s, 13m15s, 12m22s
No idea what causes that degree of variability. I guess the effect of
neighboring tasks is the most likely explanation...
In contrast, meson-64 has these "Test world" times for the same commits:
19m46s, 19m40s, 18m38s, 18m14s, 19m15s, 18m23s
(meson-64 is slow due to asan)
With the configuration for meson-64 instead done via:
- name: 'configure extra opts'
run: |
echo "io_method=io_uring" >> src/tools/ci/pg_ci_base.conf
cat src/tools/ci/pg_ci_base.conf
I see ~13m30s for meson-64.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 14+ messages in thread
* Re: CI slowdown due to PG_TEST_INITDB_EXTRA_OPTS
@ 2026-07-22 06:56 Nazir Bilal Yavuz <byavuz81@gmail.com>
parent: Andres Freund <andres@anarazel.de>
1 sibling, 1 reply; 14+ messages in thread
From: Nazir Bilal Yavuz @ 2026-07-22 06:56 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Jelte Fennema-Nio <postgres@jeltef.nl>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-hackers@postgresql.org, Peter Eisentraut <peter.eisentraut@enterprisedb.com>
Hi,
On Tue, 21 Jul 2026 at 03:56, Andres Freund <andres@anarazel.de> wrote:
> On 2026-07-20 23:47:16 +0200, Jelte Fennema-Nio wrote:
> > On Sat, 18 Jul 2026 at 20:48, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> > > It looks like any one job mostly uses the same value of
> > > PG_TEST_INITDB_EXTRA_OPTS. Could we record what was used to set up
> > > the template, and allow using it if that matches?
>
> I wonder if the best thing would be to add PG_TEST_SERVER_POSTGRES_OPTS or
> such, which Cluster.pm and regress.c would add to postgresql.conf.
Attached is an attempt to implement 'PG_TEST_SERVER_POSTGRES_OPTS'.
Like Jelte, I didn't see any improvement for MacOS.
--
Regards,
Nazir Bilal Yavuz
Microsoft
Attachments:
[text/x-patch] v2-0001-Add-PG_TEST_SERVER_POSTGRES_OPTS-to-inject-server.patch (6.0K, ../../CAN55FZ3jqqX9Th23XmDv6U-xwJjzsuc1JffYmswFT=oqy8JfrQ@mail.gmail.com/2-v2-0001-Add-PG_TEST_SERVER_POSTGRES_OPTS-to-inject-server.patch)
download | inline diff:
From 81d55a818799816c45782afa86b6c8b4e23a2fbe Mon Sep 17 00:00:00 2001
From: Nazir Bilal Yavuz <byavuz81@gmail.com>
Date: Tue, 21 Jul 2026 17:42:06 +0300
Subject: [PATCH v2] Add PG_TEST_SERVER_POSTGRES_OPTS to inject server GUCs in
tests
Unlike PG_TEST_INITDB_EXTRA_OPTS, this only appends name=value settings
to postgresql.conf without disabling the initdb template mechanism.
This speeds up CI, which sets such options in most jobs and previously
paid the full initdb cost per test cluster.
---
.github/workflows/pg-ci.yml | 16 ++++++++--------
doc/src/sgml/regress.sgml | 10 ++++++++++
src/test/perl/PostgreSQL/Test/Cluster.pm | 18 ++++++++++++++++++
src/test/regress/pg_regress.c | 24 ++++++++++++++++++++++++
4 files changed, 60 insertions(+), 8 deletions(-)
diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml
index a2629c8335a..88c12c3b35f 100644
--- a/.github/workflows/pg-ci.yml
+++ b/.github/workflows/pg-ci.yml
@@ -488,7 +488,7 @@ jobs:
echo "CXX=${CXX}" >> "$GITHUB_ENV"
echo "PG_TEST_PG_UPGRADE_MODE=${PG_TEST_PG_UPGRADE_MODE}" >> "$GITHUB_ENV"
- echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV"
+ echo "PG_TEST_SERVER_POSTGRES_OPTS=${PG_TEST_SERVER_POSTGRES_OPTS}" >> "$GITHUB_ENV"
echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV"
- *nix_sysinfo_step
@@ -548,7 +548,7 @@ jobs:
- name: Update Environment
env:
SANITIZER_FLAGS: -fsanitize=alignment,undefined
- PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring
+ PG_TEST_SERVER_POSTGRES_OPTS: io_method=io_uring
CC: ccache gcc -m32
CXX: ccache g++ -m32
run: *linux_update_config_cmd
@@ -640,7 +640,7 @@ jobs:
- name: Update Environment
env:
SANITIZER_FLAGS: -fsanitize=address
- PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring
+ PG_TEST_SERVER_POSTGRES_OPTS: io_method=io_uring
run: *linux_update_config_cmd
- *nix_sysinfo_step
@@ -724,11 +724,11 @@ jobs:
# Several buildfarm animals enable these options. Without testing them
# during CI, it would be easy to cause breakage on the buildfarm with CI
# passing.
- PG_TEST_INITDB_EXTRA_OPTS: >-
- -c debug_copy_parse_plan_trees=on
- -c debug_write_read_parse_plan_trees=on
- -c debug_raw_expression_coverage_test=on
- -c debug_parallel_query=regress
+ PG_TEST_SERVER_POSTGRES_OPTS: >-
+ debug_copy_parse_plan_trees=on
+ debug_write_read_parse_plan_trees=on
+ debug_raw_expression_coverage_test=on
+ debug_parallel_query=regress
steps:
- *nix_sysinfo_step
diff --git a/doc/src/sgml/regress.sgml b/doc/src/sgml/regress.sgml
index c74941bfbf2..be121a4ab90 100644
--- a/doc/src/sgml/regress.sgml
+++ b/doc/src/sgml/regress.sgml
@@ -554,6 +554,16 @@ make check PG_TEST_INITDB_EXTRA_OPTS='-k --wal-segsize=4 -c work_mem=50MB'
</screen>
</para>
+ <para>
+ Run-time server settings can be added to the
+ <filename>postgresql.conf</filename> using the environment variable
+ <envar>PG_TEST_SERVER_POSTGRES_OPTS</envar>. The settings are given as
+ whitespace-separated <literal>name=value</literal> pairs. For example:
+<screen>
+make check PG_TEST_SERVER_POSTGRES_OPTS='work_mem=50MB debug_discard_caches=1'
+</screen>
+ </para>
+
<para>
For the core regression test suite and other tests driven by
<command>pg_regress</command>, custom run-time server settings can also be
diff --git a/src/test/perl/PostgreSQL/Test/Cluster.pm b/src/test/perl/PostgreSQL/Test/Cluster.pm
index 3eae4cf6281..7a29cb84efc 100644
--- a/src/test/perl/PostgreSQL/Test/Cluster.pm
+++ b/src/test/perl/PostgreSQL/Test/Cluster.pm
@@ -749,6 +749,24 @@ sub init
print $conf "max_wal_senders = 0\n";
}
+ # PG_TEST_SERVER_POSTGRES_OPTS allows adding server settings via an
+ # environment variable. Unlike PG_TEST_INITDB_EXTRA_OPTS, it does not
+ # disable the initdb template mechanism, as the options are only added to
+ # postgresql.conf and not passed to initdb. The options are given as
+ # whitespace-separated "name=value" pairs.
+ #
+ # There's very similar code in pg_regress.c.
+ my $server_opts_env = $ENV{PG_TEST_SERVER_POSTGRES_OPTS};
+ if (defined $server_opts_env)
+ {
+ print $conf "# options from PG_TEST_SERVER_POSTGRES_OPTS\n";
+ foreach my $opt (split(/\s+/, $server_opts_env))
+ {
+ print $conf "$opt\n" if $opt ne '';
+ }
+ print $conf "# end of options from PG_TEST_SERVER_POSTGRES_OPTS\n";
+ }
+
print $conf "port = $port\n";
if ($use_tcp)
{
diff --git a/src/test/regress/pg_regress.c b/src/test/regress/pg_regress.c
index 21d00f792d2..d6c1521bfaf 100644
--- a/src/test/regress/pg_regress.c
+++ b/src/test/regress/pg_regress.c
@@ -2372,6 +2372,7 @@ regression_main(int argc, char *argv[],
const char *values[4];
PGPing rv;
const char *initdb_extra_opts_env;
+ const char *server_opts_env;
/*
* Prepare the temp instance
@@ -2500,6 +2501,29 @@ regression_main(int argc, char *argv[],
fclose(extra_conf);
}
+ /*
+ * PG_TEST_SERVER_POSTGRES_OPTS allows adding server settings via an
+ * environment variable. Unlike PG_TEST_INITDB_EXTRA_OPTS, it does
+ * not disable the initdb template mechanism, as the options are only
+ * added to postgresql.conf and not passed to initdb. The options are
+ * given as whitespace-separated "name=value" pairs.
+ *
+ * There's very similar code in Cluster.pm.
+ */
+ server_opts_env = getenv("PG_TEST_SERVER_POSTGRES_OPTS");
+ if (server_opts_env != NULL)
+ {
+ char *opts = pg_strdup(server_opts_env);
+ char *tok;
+
+ fputs("# options from PG_TEST_SERVER_POSTGRES_OPTS\n", pg_conf);
+ for (tok = strtok(opts, " \t\n"); tok != NULL;
+ tok = strtok(NULL, " \t\n"))
+ fprintf(pg_conf, "%s\n", tok);
+ free(opts);
+ fputs("# end of options from PG_TEST_SERVER_POSTGRES_OPTS\n", pg_conf);
+ }
+
fclose(pg_conf);
#ifdef ENABLE_SSPI
--
2.47.3
^ permalink raw reply [nested|flat] 14+ messages in thread
* Re: CI slowdown due to PG_TEST_INITDB_EXTRA_OPTS
@ 2026-08-11 12:39 Jelte Fennema-Nio <postgres@jeltef.nl>
parent: Andres Freund <andres@anarazel.de>
1 sibling, 2 replies; 14+ messages in thread
From: Jelte Fennema-Nio @ 2026-08-11 12:39 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; pgsql-hackers@postgresql.org, Peter Eisentraut <peter.eisentraut@enterprisedb.com>; Nazir Bilal Yavuz <byavuz81@gmail.com>
On Tue, 21 Jul 2026 at 02:56, Andres Freund <andres@anarazel.de> wrote:
>
> Hi,
>
> On 2026-07-20 23:47:16 +0200, Jelte Fennema-Nio wrote:
> > On Sat, 18 Jul 2026 at 20:48, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> > > It looks like any one job mostly uses the same value of
> > > PG_TEST_INITDB_EXTRA_OPTS. Could we record what was used to set up
> > > the template, and allow using it if that matches?
>
> That seems like a somewhat complicated mechanism. But mainly I feel like it's
> a workaround around the config-customization-via-env-var having to go through
> exactly one file. None of the uses of PG_TEST_INITDB_EXTRA_OPTS in CI actually
> needs it to be an initdb option, it's just a way to set configuration options
> for the server via an environment variable.
>
> I wonder if the best thing would be to add PG_TEST_SERVER_POSTGRES_OPTS or
> such, which Cluster.pm and regress.c would add to postgresql.conf.
I don't think it's that complicated. But I'd be fine with
PG_TEST_SERVER_POSTGRES_OPTS as well. That has the downside that it
won't work for actual initdb options, but at least CI doesn't use that
(for now). Some buildfarm members might though.
^ permalink raw reply [nested|flat] 14+ messages in thread
* Re: CI slowdown due to PG_TEST_INITDB_EXTRA_OPTS
@ 2026-08-13 10:09 Jakub Wartak <jakub.wartak@enterprisedb.com>
parent: Jelte Fennema-Nio <postgres@jeltef.nl>
1 sibling, 0 replies; 14+ messages in thread
From: Jakub Wartak @ 2026-08-13 10:09 UTC (permalink / raw)
To: Jelte Fennema-Nio <postgres@jeltef.nl>; +Cc: Andres Freund <andres@anarazel.de>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-hackers@postgresql.org, Peter Eisentraut <peter.eisentraut@enterprisedb.com>; Nazir Bilal Yavuz <byavuz81@gmail.com>
On Tue, Aug 11, 2026 at 2:39 PM Jelte Fennema-Nio <postgres@jeltef.nl> wrote:
>
> On Tue, 21 Jul 2026 at 02:56, Andres Freund <andres@anarazel.de> wrote:
> >
> > Hi,
> >
> > On 2026-07-20 23:47:16 +0200, Jelte Fennema-Nio wrote:
> > > On Sat, 18 Jul 2026 at 20:48, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> > > > It looks like any one job mostly uses the same value of
> > > > PG_TEST_INITDB_EXTRA_OPTS. Could we record what was used to set up
> > > > the template, and allow using it if that matches?
> >
> > That seems like a somewhat complicated mechanism. But mainly I feel like it's
> > a workaround around the config-customization-via-env-var having to go through
> > exactly one file. None of the uses of PG_TEST_INITDB_EXTRA_OPTS in CI actually
> > needs it to be an initdb option, it's just a way to set configuration options
> > for the server via an environment variable.
> >
> > I wonder if the best thing would be to add PG_TEST_SERVER_POSTGRES_OPTS or
> > such, which Cluster.pm and regress.c would add to postgresql.conf.
>
> I don't think it's that complicated. But I'd be fine with
> PG_TEST_SERVER_POSTGRES_OPTS as well. That has the downside that it
> won't work for actual initdb options, but at least CI doesn't use that
> (for now). Some buildfarm members might though.
Hi,
I've scheduled 3 runs on 3 consecutive days (w/o and with v2 patch by Nazir),
[in minutes, from/master -> branch/w patch]
autoconf 23 -> 17, 21 -> 10, 20 -> 16
meson32 16 -> 14, 16 -> 9, 15 -> 13
meson64 29 -> 22, 28 -> 18, 28 -> 22
macOS 28 -> 25, 21 -> 17, 21 -> 20
mingw 22 -> 27?,25 -> 23, 28 -> 26
warnings 32 -> 30, 23 -> 3?, 27 -> 24
winvc1 18 -> 19, 19 -> 26, 16 -> 16
winvc2 17 -> 17, 17 -> 14, 17 -> 16
Take the results with some degree of doubt (I have no idea what is
infrastructure underneath; how overscribed the CPUs were etc.), but those
are consistent, real gains for Linux builds (-33% autoconf, -27% meson32,
23% meson64) during 3 days.
> I was once more wondering why CI is so slow. [..]
Cheer up gents, altough it's like ~ < 25min, I find the full CI of PostgreSQL
still the fastest thing I have ever worked with and it's free. From what
I've worked with, heard or read in terms of complex software/systems it's
leaving the rest of world in the dust in terms of how lightweight/fast it
is ... [one can have it even faster on your hardware if one wants -- using
runners].
-J.
^ permalink raw reply [nested|flat] 14+ messages in thread
* Re: CI slowdown due to PG_TEST_INITDB_EXTRA_OPTS
@ 2026-08-13 15:37 Greg Sabino Mullane <htamfids@gmail.com>
parent: Jelte Fennema-Nio <postgres@jeltef.nl>
1 sibling, 1 reply; 14+ messages in thread
From: Greg Sabino Mullane @ 2026-08-13 15:37 UTC (permalink / raw)
To: Jelte Fennema-Nio <postgres@jeltef.nl>; +Cc: Andres Freund <andres@anarazel.de>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-hackers@postgresql.org, Peter Eisentraut <peter.eisentraut@enterprisedb.com>; Nazir Bilal Yavuz <byavuz81@gmail.com>
On Tue, Aug 11, 2026 at 8:39 AM Jelte Fennema-Nio <postgres@jeltef.nl>
wrote:
> I don't think it's that complicated. But I'd be fine with
> PG_TEST_SERVER_POSTGRES_OPTS as well. That has the downside that it
> won't work for actual initdb options, but at least CI doesn't use that
> (for now). Some buildfarm members might though.
>
But they can still use PG_TEST_INITDB_EXTRA_OPTS, right? We are not
replacing the var, just adding a new one, I thought. Probably should
eventually look into caching things for people that use true initdb
options, but I suspect the number of things not handled by the new var will
be very close to zero, so maybe not worth it.
Cheers,
Greg
^ permalink raw reply [nested|flat] 14+ messages in thread
* Re: CI slowdown due to PG_TEST_INITDB_EXTRA_OPTS
@ 2026-09-06 18:12 Andrey Borodin <x4mmm@yandex-team.ru>
parent: Greg Sabino Mullane <htamfids@gmail.com>
0 siblings, 1 reply; 14+ messages in thread
From: Andrey Borodin @ 2026-09-06 18:12 UTC (permalink / raw)
To: Greg Sabino Mullane <htamfids@gmail.com>; +Cc: Jelte Fennema-Nio <postgres@jeltef.nl>; Andres Freund <andres@anarazel.de>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-hackers@postgresql.org, Peter Eisentraut <peter.eisentraut@enterprisedb.com>; Nazir Bilal Yavuz <byavuz81@gmail.com>
On Thu, Aug 13, 2026, Greg Sabino Mullane wrote:
> But they can still use PG_TEST_INITDB_EXTRA_OPTS, right? We are not
> replacing the var, just adding a new one, I thought.
Yes. I read Nazir's v2. It keeps PG_TEST_INITDB_EXTRA_OPTS
available for actual initdb options. Both Cluster.pm and pg_regress
handle the new variable consistently, and the CI changes move only
server GUCs to it. The patch LGTM.
I ran CI quite a few times today and spent some time waiting for it,
so I started looking for ways to make it faster.
I also tried moving one-use pg_rewind backups instead of copying them,
and caching the dummy_seclabel availability check across pg_upgrade
transfer modes. They save a few percent in the affected tests. I can
send those patches too if that sounds worthwhile.
The lowest-hanging fruit I found is splitting the Linux Meson 64-bit
ASAN tests across two runners, as we already do for the Windows Visual
Studio job. This does not reduce the total work and actually
duplicates the configure and build steps, but it does reduce how long
we wait for CI.
In one run, the two Test world slices took 8:33 and 12:08. Two
unsliced jobs running at the same time took 23:45 and 25:05. Together
the slices ran the same 412 tests. In a complete run, the gain is
naturally limited by the next-slowest job.
The CI image currently has Meson 1.7, so the patch emulates Meson 1.8's
round-robin --slice using the test list.
PFA. WDYT?
Best regards, Andrey Borodin.
Attachments:
[application/octet-stream] v1-0001-Split-Linux-ASAN-tests-across-two-runners.patch (3.7K, ../../EDB6D574-061F-4E58-B927-383884C35EBE@yandex-team.ru/2-v1-0001-Split-Linux-ASAN-tests-across-two-runners.patch)
download | inline diff:
From 84455086f3cf088375a8d8b4ff31a73329fb795c Mon Sep 17 00:00:00 2001
From: Andrey Borodin <amborodin@acm.org>
Date: Sun, 6 Sep 2026 21:38:01 +0500
Subject: [PATCH v1] Split Linux ASAN tests across two runners
The Linux Meson 64-bit job often determines CI latency because its
ASAN test suite takes much longer than its cached build and generally
longer than other jobs.
Run the setup suite on both runners and divide the remaining Meson tests
round-robin. The CI image has Meson 1.7, so derive the slices from
"meson test --list" instead of using the Meson 1.8 --slice option.
Use separate ccache save keys and failure-log artifact names for the
matrix jobs.
ci-os-only: linux
---
.github/workflows/pg-ci.yml | 51 ++++++++++++++++++++++++++++++++++---
1 file changed, 48 insertions(+), 3 deletions(-)
diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml
index a2629c8335a..8c13baf36b0 100644
--- a/.github/workflows/pg-ci.yml
+++ b/.github/workflows/pg-ci.yml
@@ -625,10 +625,11 @@ jobs:
# - Uses address sanitizer, (sanitizer failures are typically printed in the
# server log). We test asan with meson rather than autoconf, as it's a bit
# faster at running the tests.
+ # - Splits the tests across two runners
# - Uses io_method=io_uring
# - Uses meson feature autodetection
linux-meson-64:
- name: Linux - Meson (64-bit)
+ name: Linux - Meson (64-bit) - Slice ${{ matrix.slice}}/${{ matrix.num_slices}}
needs: [setup, sanity-check]
if: *linux_job_if
runs-on: *linux_runs_on
@@ -636,6 +637,12 @@ jobs:
timeout-minutes: 60
env: *linux_env
+ strategy:
+ fail-fast: false
+ matrix:
+ num_slices: [2]
+ slice: [1, 2]
+
steps:
- name: Update Environment
env:
@@ -643,6 +650,9 @@ jobs:
PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring
run: *linux_update_config_cmd
+ - name: Distinguish ccache slice
+ run: echo "CACHE_SUFFIX=${CACHE_SUFFIX}:${{ matrix.slice }}" >> "$GITHUB_ENV"
+
- *nix_sysinfo_step
- *checkout_step
- *ccache_restore_default_step
@@ -668,10 +678,45 @@ jobs:
- name: Test world
shell: *su_postgres_shell
- run: *meson_test_world_cmd
+ run: |
+ ulimit -c unlimited
+
+ echo ::group::test_setup
+ meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1
+ echo ::endgroup::
+
+ # The system Meson is older than 1.8, which introduced --slice.
+ # Emulate its round-robin slicing using the test list instead.
+ readarray -t test_names < <(
+ meson test ${{env.MTEST_ARGS}} ${{env.MTEST_TARGET}} \
+ --list --no-suite setup |
+ awk -F ' - ' \
+ -v slice=${{ matrix.slice}} \
+ -v slices=${{ matrix.num_slices}} \
+ '(NR - 1) % slices == slice - 1 {
+ name = $NF
+ sub(/:[^ ]+ \/ /, ":", name)
+ print name
+ }'
+ )
+ meson test ${{env.MTEST_ARGS}} \
+ --num-processes ${{env.TEST_JOBS}} \
+ --no-suite setup "${test_names[@]}"
- *linux_collect_cores_step
- - *upload_logs_step
+ - name: Upload logs
+ if: failure() && !cancelled()
+ uses: actions/upload-artifact@v7
+ with:
+ name: logs-${{ github.job }}-${{ matrix.slice }}-${{ github.run_id }}-${{ github.run_attempt }}
+ path: |
+ **/*.log
+ **/*.diffs
+ **/regress_log_*
+ **/crashlog-*.txt
+ build/meson-logs/**
+ **/config.log
+ if-no-files-found: ignore
# Job: macOS - Meson
--
That's all, folks. May the source be with you.
=
^ permalink raw reply [nested|flat] 14+ messages in thread
* Re: CI slowdown due to PG_TEST_INITDB_EXTRA_OPTS
@ 2026-09-07 11:06 Nazir Bilal Yavuz <byavuz81@gmail.com>
parent: Andrey Borodin <x4mmm@yandex-team.ru>
0 siblings, 1 reply; 14+ messages in thread
From: Nazir Bilal Yavuz @ 2026-09-07 11:06 UTC (permalink / raw)
To: Andrey Borodin <x4mmm@yandex-team.ru>; +Cc: Greg Sabino Mullane <htamfids@gmail.com>; Jelte Fennema-Nio <postgres@jeltef.nl>; Andres Freund <andres@anarazel.de>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-hackers@postgresql.org, Peter Eisentraut <peter.eisentraut@enterprisedb.com>
Hi,
On Sun, 6 Sept 2026 at 21:12, Andrey Borodin <x4mmm@yandex-team.ru> wrote:
>
> On Thu, Aug 13, 2026, Greg Sabino Mullane wrote:
> > But they can still use PG_TEST_INITDB_EXTRA_OPTS, right? We are not
> > replacing the var, just adding a new one, I thought.
>
> Yes. I read Nazir's v2. It keeps PG_TEST_INITDB_EXTRA_OPTS
> available for actual initdb options. Both Cluster.pm and pg_regress
> handle the new variable consistently, and the CI changes move only
> server GUCs to it. The patch LGTM.
Thank you for looking into this!
>
> The lowest-hanging fruit I found is splitting the Linux Meson 64-bit
> ASAN tests across two runners, as we already do for the Windows Visual
> Studio job. This does not reduce the total work and actually
> duplicates the configure and build steps, but it does reduce how long
> we wait for CI.
>
> In one run, the two Test world slices took 8:33 and 12:08. Two
> unsliced jobs running at the same time took 23:45 and 25:05. Together
> the slices ran the same 412 tests. In a complete run, the gain is
> naturally limited by the next-slowest job.
>
> The CI image currently has Meson 1.7, so the patch emulates Meson 1.8's
> round-robin --slice using the test list.
First question that comes to mind is why we are doing this only for
Linux 64 Meson and not for all jobs. I think there are two answers:
1. We can't do this for Linux Autoconf as far as I know, so it doesn't
make sense for tasks that take less or similar time.
2. The free concurrency limit for GitHub Actions is 20, and our main
CI tasks use 8 (9 with your patch). It would be best to not exceed 10
so that we can run two CI tasks simultaneously.
So, I think this change makes sense. Also we can apply the same change
to the next longest running task, which is MinGW 64 depending on my
testing.
My review:
+ readarray -t test_names < <(
+ meson test ${{env.MTEST_ARGS}} ${{env.MTEST_TARGET}} \
+ --list --no-suite setup |
+ awk -F ' - ' \
+ -v slice=${{ matrix.slice}} \
+ -v slices=${{ matrix.num_slices}} \
+ '(NR - 1) % slices == slice - 1 {
+ name = $NF
+ sub(/:[^ ]+ \/ /, ":", name)
+ print name
+ }'
+ )
This is complicated but I don't have a better solution. One point is
that 'meson test --list' doesn't guarantee the order of the returned
list, so we might miss tests if the order differs between tasks. I
recommend sorting the list names before slicing.
- - *upload_logs_step
+ - name: Upload logs
+ if: failure() && !cancelled()
+ uses: actions/upload-artifact@v7
+ with:
+ name: logs-${{ github.job }}-${{ matrix.slice }}-${{
github.run_id }}-${{ github.run_attempt }}
+ path: |
+ **/*.log
+ **/*.diffs
+ **/regress_log_*
+ **/crashlog-*.txt
+ build/meson-logs/**
+ **/config.log
+ if-no-files-found: ignore
We have the same problem for the Windows VS job; otherwise, two
artifacts could end up having the same name. I remember sending a
patch to fix this, but I don't recall where it is. I think we can save
this step as something like 'upload_logs_step_slice' and use it in the
Windows VS task as well.
--
Regards,
Nazir Bilal Yavuz
Microsoft
^ permalink raw reply [nested|flat] 14+ messages in thread
* Re: CI slowdown due to PG_TEST_INITDB_EXTRA_OPTS
@ 2026-09-07 13:15 Andrey Borodin <x4mmm@yandex-team.ru>
parent: Nazir Bilal Yavuz <byavuz81@gmail.com>
0 siblings, 1 reply; 14+ messages in thread
From: Andrey Borodin @ 2026-09-07 13:15 UTC (permalink / raw)
To: Nazir Bilal Yavuz <byavuz81@gmail.com>; +Cc: Greg Sabino Mullane <htamfids@gmail.com>; Jelte Fennema-Nio <postgres@jeltef.nl>; Andres Freund <andres@anarazel.de>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-hackers; Peter Eisentraut <peter.eisentraut@enterprisedb.com>
Hi Nazir,
Thank you for the review.
> 'meson test --list' doesn't guarantee the order of the returned
> list, so we might miss tests if the order differs between tasks.
Agreed. Since the two runners build their lists independently, a
different order could produce both a missing test and a duplicate. V2
sorts the names before applying the round-robin selection.
> I think we can save this step as something like
> 'upload_logs_step_slice' and use it in the Windows VS task as well.
Agreed. V2 defines the slice-aware upload step in the Linux job and
reuses it for the existing Windows Visual Studio matrix. This also
avoids the existing artifact-name collision when both Windows slices
fail.
I also tried splitting MinGW. In six recent full runs the unsliced job
took 28:36-31:29 and was the slowest job each time. The two slices
completed in 15:05 and 17:32, so v2 includes this too.
PFA v2.
Best regards, Andrey Borodin.
Attachments:
[application/octet-stream] v2-0001-Split-Linux-ASAN-and-MinGW-tests-across-runners.patch (5.5K, ../../0C9F9047-0C12-49E4-9954-8BCA6B861C75@yandex-team.ru/2-v2-0001-Split-Linux-ASAN-and-MinGW-tests-across-runners.patch)
download | inline diff:
From 2789d582049ea60cb14e882c4cc3d968e07f71a1 Mon Sep 17 00:00:00 2001
From: Andrey Borodin <amborodin@acm.org>
Date: Sun, 6 Sep 2026 21:38:01 +0500
Subject: [PATCH v2] Split Linux ASAN and MinGW tests across runners
The Linux Meson 64-bit and Windows MinGW jobs often determine CI
latency because their test suites take much longer than their cached
builds.
Split each test suite across two runners while running the setup suite
on both. Use Meson's native --slice support for MinGW. The Linux CI
image has Meson 1.7, so derive a deterministically sorted test list
there instead of using the Meson 1.8 --slice option.
Use separate ccache save keys and failure-log artifact names for matrix
jobs. Share the slice-specific upload step with the existing Windows
Visual Studio matrix, whose jobs otherwise use the same artifact name.
ci-os-only: linux windows mingw
---
.github/workflows/pg-ci.yml | 73 ++++++++++++++++++++++++++++++++++---
1 file changed, 67 insertions(+), 6 deletions(-)
diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml
index a2629c8335a..6ec10fabca1 100644
--- a/.github/workflows/pg-ci.yml
+++ b/.github/workflows/pg-ci.yml
@@ -625,10 +625,11 @@ jobs:
# - Uses address sanitizer, (sanitizer failures are typically printed in the
# server log). We test asan with meson rather than autoconf, as it's a bit
# faster at running the tests.
+ # - Splits the tests across two runners
# - Uses io_method=io_uring
# - Uses meson feature autodetection
linux-meson-64:
- name: Linux - Meson (64-bit)
+ name: Linux - Meson (64-bit) - Slice ${{ matrix.slice}}/${{ matrix.num_slices}}
needs: [setup, sanity-check]
if: *linux_job_if
runs-on: *linux_runs_on
@@ -636,6 +637,12 @@ jobs:
timeout-minutes: 60
env: *linux_env
+ strategy:
+ fail-fast: false
+ matrix:
+ num_slices: [2]
+ slice: [1, 2]
+
steps:
- name: Update Environment
env:
@@ -643,6 +650,9 @@ jobs:
PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring
run: *linux_update_config_cmd
+ - name: Distinguish ccache slice
+ run: echo "CACHE_SUFFIX=${CACHE_SUFFIX}:${{ matrix.slice }}" >> "$GITHUB_ENV"
+
- *nix_sysinfo_step
- *checkout_step
- *ccache_restore_default_step
@@ -668,10 +678,47 @@ jobs:
- name: Test world
shell: *su_postgres_shell
- run: *meson_test_world_cmd
+ run: |
+ ulimit -c unlimited
+
+ echo ::group::test_setup
+ meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1
+ echo ::endgroup::
+
+ # The system Meson is older than 1.8, which introduced --slice.
+ # Emulate its round-robin slicing using the test list instead.
+ readarray -t test_names < <(
+ meson test ${{env.MTEST_ARGS}} ${{env.MTEST_TARGET}} \
+ --list --no-suite setup |
+ sort |
+ awk -F ' - ' \
+ -v slice=${{ matrix.slice}} \
+ -v slices=${{ matrix.num_slices}} \
+ '(NR - 1) % slices == slice - 1 {
+ name = $NF
+ sub(/:[^ ]+ \/ /, ":", name)
+ print name
+ }'
+ )
+ meson test ${{env.MTEST_ARGS}} \
+ --num-processes ${{env.TEST_JOBS}} \
+ --no-suite setup "${test_names[@]}"
- *linux_collect_cores_step
- - *upload_logs_step
+ - &upload_logs_sliced_step
+ name: Upload logs
+ if: failure() && !cancelled()
+ uses: actions/upload-artifact@v7
+ with:
+ name: logs-${{ github.job }}-${{ matrix.slice }}-${{ github.run_id }}-${{ github.run_attempt }}
+ path: |
+ **/*.log
+ **/*.diffs
+ **/regress_log_*
+ **/crashlog-*.txt
+ build/meson-logs/**
+ **/config.log
+ if-no-files-found: ignore
# Job: macOS - Meson
@@ -1024,12 +1071,12 @@ jobs:
call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64
run: *meson_test_world_cmd
- - *upload_logs_step
+ - *upload_logs_sliced_step
# Job: Windows - MinGW - Meson
windows-mingw:
- name: Windows - MinGW - Meson
+ name: Windows - MinGW - Meson - Slice ${{ matrix.slice}}/${{ matrix.num_slices}}
needs: [setup, sanity-check]
if: |
!cancelled() &&
@@ -1037,6 +1084,15 @@ jobs:
needs.sanity-check.result != 'failure'
runs-on: windows-2022
timeout-minutes: 60
+
+ # MinGW is generally one of the slowest CI jobs. Split its tests across
+ # two runners, as is already done for the Visual Studio job.
+ strategy:
+ fail-fast: false
+ matrix:
+ num_slices: [2]
+ slice: [1, 2]
+
env:
# Avoid port conflicts between concurrent tap tests
PG_TEST_USE_UNIX_SOCKETS: 1
@@ -1120,6 +1176,9 @@ jobs:
- *windows_setup_debugger_step
+ - name: Distinguish ccache slice
+ run: echo "CACHE_SUFFIX=${CACHE_SUFFIX}:${{ matrix.slice }}" >> "$GITHUB_ENV"
+
- *ccache_restore_default_step
- *ccache_restore_branch_step
@@ -1140,9 +1199,11 @@ jobs:
- *ccache_save_step
- name: Test world
+ env:
+ MTEST_TARGET: --slice ${{ matrix.slice}}/${{ matrix.num_slices}} ${{env.MTEST_TARGET}}
run: *meson_test_world_cmd
- - *upload_logs_step
+ - *upload_logs_sliced_step
# Job: CompilerWarnings
--
That's all, folks. May the source be with you.
=
^ permalink raw reply [nested|flat] 14+ messages in thread
* Re: CI slowdown due to PG_TEST_INITDB_EXTRA_OPTS
@ 2026-09-10 11:45 Nazir Bilal Yavuz <byavuz81@gmail.com>
parent: Andrey Borodin <x4mmm@yandex-team.ru>
0 siblings, 1 reply; 14+ messages in thread
From: Nazir Bilal Yavuz @ 2026-09-10 11:45 UTC (permalink / raw)
To: Andrey Borodin <x4mmm@yandex-team.ru>; +Cc: Greg Sabino Mullane <htamfids@gmail.com>; Jelte Fennema-Nio <postgres@jeltef.nl>; Andres Freund <andres@anarazel.de>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-hackers; Peter Eisentraut <peter.eisentraut@enterprisedb.com>
Hi,
On Mon, 7 Sept 2026 at 16:15, Andrey Borodin <x4mmm@yandex-team.ru> wrote:
>
> Agreed. V2 defines the slice-aware upload step in the Linux job and
> reuses it for the existing Windows Visual Studio matrix. This also
> avoids the existing artifact-name collision when both Windows slices
> fail.
>
> I also tried splitting MinGW. In six recent full runs the unsliced job
> took 28:36-31:29 and was the slowest job each time. The two slices
> completed in 15:05 and 17:32, so v2 includes this too.
+ - name: Distinguish ccache slice
+ run: echo "CACHE_SUFFIX=${CACHE_SUFFIX}:${{ matrix.slice }}"
>> "$GITHUB_ENV"
I would remove these steps; ccache don't differ between builds so
there is no need to separate them. Other than that, LGTM.
--
Regards,
Nazir Bilal Yavuz
Microsoft
^ permalink raw reply [nested|flat] 14+ messages in thread
* Re: CI slowdown due to PG_TEST_INITDB_EXTRA_OPTS
@ 2026-09-11 06:29 Andrey Borodin <x4mmm@yandex-team.ru>
parent: Nazir Bilal Yavuz <byavuz81@gmail.com>
0 siblings, 0 replies; 14+ messages in thread
From: Andrey Borodin @ 2026-09-11 06:29 UTC (permalink / raw)
To: Nazir Bilal Yavuz <byavuz81@gmail.com>; +Cc: Greg Sabino Mullane <htamfids@gmail.com>; Jelte Fennema-Nio <postgres@jeltef.nl>; Andres Freund <andres@anarazel.de>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-hackers; Peter Eisentraut <peter.eisentraut@enterprisedb.com>
Hi Nazir,
On Thu, Sep 10, 2026, Nazir Bilal Yavuz wrote:
> I would remove these steps; ccache don't differ between builds so
> there is no need to separate them. Other than that, LGTM.
Agreed. V3 removes the slice suffix from the ccache keys for both
Linux ASAN and MinGW.
For context, here are the timings from v3 run, without your
PG_TEST_SERVER_POSTGRES_OPTS patch.
Test world Complete job
Linux Autoconf 15:04 22:00
Linux Meson 32-bit 9:53 16:29
Linux ASAN slice 1/2 12:05 17:17
Linux ASAN slice 2/2 11:52 17:04
macOS 20:29 24:29
MinGW slice 1/2 8:51 19:13
MinGW slice 2/2 7:29 17:57
Visual Studio slice 1/2 13:01 16:48
Visual Studio slice 2/2 11:45 16:33
CompilerWarnings - 22:13
Further splitting any job would not help much.
Your patch is still much needed: it reduces the work itself.
Thank you for the review!
PFA v3.
Best regards, Andrey Borodin.
Attachments:
[application/octet-stream] v3-0001-Split-Linux-ASAN-and-MinGW-tests-across-runners.patch (4.8K, ../../1820B28B-6BDA-4815-AD0B-9BBAB1D3A820@yandex-team.ru/2-v3-0001-Split-Linux-ASAN-and-MinGW-tests-across-runners.patch)
download | inline diff:
From 67c36884bc279742abc19069972a784bb7629f12 Mon Sep 17 00:00:00 2001
From: Andrey Borodin <amborodin@acm.org>
Date: Thu, 10 Sep 2026 18:17:00 +0500
Subject: [PATCH v3] Split Linux ASAN and MinGW tests across runners
The Linux Meson 64-bit and Windows MinGW jobs often determine CI
latency because their test suites take much longer than their cached
builds.
Split each test suite across two runners while running the setup suite
on both. Use Meson's native --slice support for MinGW. The Linux CI
image has Meson 1.7, so derive a deterministically sorted test list
there instead of using the Meson 1.8 --slice option.
Use separate failure-log artifact names for matrix jobs. Share the
slice-specific upload step with the existing Windows Visual Studio
matrix, whose jobs otherwise use the same artifact name.
---
.github/workflows/pg-ci.yml | 67 +++++++++++++++++++++++++++++++++----
1 file changed, 61 insertions(+), 6 deletions(-)
diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml
index a2629c8335a..f0859b245fb 100644
--- a/.github/workflows/pg-ci.yml
+++ b/.github/workflows/pg-ci.yml
@@ -625,10 +625,11 @@ jobs:
# - Uses address sanitizer, (sanitizer failures are typically printed in the
# server log). We test asan with meson rather than autoconf, as it's a bit
# faster at running the tests.
+ # - Splits the tests across two runners
# - Uses io_method=io_uring
# - Uses meson feature autodetection
linux-meson-64:
- name: Linux - Meson (64-bit)
+ name: Linux - Meson (64-bit) - Slice ${{ matrix.slice}}/${{ matrix.num_slices}}
needs: [setup, sanity-check]
if: *linux_job_if
runs-on: *linux_runs_on
@@ -636,6 +637,12 @@ jobs:
timeout-minutes: 60
env: *linux_env
+ strategy:
+ fail-fast: false
+ matrix:
+ num_slices: [2]
+ slice: [1, 2]
+
steps:
- name: Update Environment
env:
@@ -668,10 +675,47 @@ jobs:
- name: Test world
shell: *su_postgres_shell
- run: *meson_test_world_cmd
+ run: |
+ ulimit -c unlimited
+
+ echo ::group::test_setup
+ meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1
+ echo ::endgroup::
+
+ # The system Meson is older than 1.8, which introduced --slice.
+ # Emulate its round-robin slicing using the test list instead.
+ readarray -t test_names < <(
+ meson test ${{env.MTEST_ARGS}} ${{env.MTEST_TARGET}} \
+ --list --no-suite setup |
+ sort |
+ awk -F ' - ' \
+ -v slice=${{ matrix.slice}} \
+ -v slices=${{ matrix.num_slices}} \
+ '(NR - 1) % slices == slice - 1 {
+ name = $NF
+ sub(/:[^ ]+ \/ /, ":", name)
+ print name
+ }'
+ )
+ meson test ${{env.MTEST_ARGS}} \
+ --num-processes ${{env.TEST_JOBS}} \
+ --no-suite setup "${test_names[@]}"
- *linux_collect_cores_step
- - *upload_logs_step
+ - &upload_logs_sliced_step
+ name: Upload logs
+ if: failure() && !cancelled()
+ uses: actions/upload-artifact@v7
+ with:
+ name: logs-${{ github.job }}-${{ matrix.slice }}-${{ github.run_id }}-${{ github.run_attempt }}
+ path: |
+ **/*.log
+ **/*.diffs
+ **/regress_log_*
+ **/crashlog-*.txt
+ build/meson-logs/**
+ **/config.log
+ if-no-files-found: ignore
# Job: macOS - Meson
@@ -1024,12 +1068,12 @@ jobs:
call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64
run: *meson_test_world_cmd
- - *upload_logs_step
+ - *upload_logs_sliced_step
# Job: Windows - MinGW - Meson
windows-mingw:
- name: Windows - MinGW - Meson
+ name: Windows - MinGW - Meson - Slice ${{ matrix.slice}}/${{ matrix.num_slices}}
needs: [setup, sanity-check]
if: |
!cancelled() &&
@@ -1037,6 +1081,15 @@ jobs:
needs.sanity-check.result != 'failure'
runs-on: windows-2022
timeout-minutes: 60
+
+ # MinGW is generally one of the slowest CI jobs. Split its tests across
+ # two runners, as is already done for the Visual Studio job.
+ strategy:
+ fail-fast: false
+ matrix:
+ num_slices: [2]
+ slice: [1, 2]
+
env:
# Avoid port conflicts between concurrent tap tests
PG_TEST_USE_UNIX_SOCKETS: 1
@@ -1140,9 +1193,11 @@ jobs:
- *ccache_save_step
- name: Test world
+ env:
+ MTEST_TARGET: --slice ${{ matrix.slice}}/${{ matrix.num_slices}} ${{env.MTEST_TARGET}}
run: *meson_test_world_cmd
- - *upload_logs_step
+ - *upload_logs_sliced_step
# Job: CompilerWarnings
--
That's all, folks. May the source be with you.
=
^ permalink raw reply [nested|flat] 14+ messages in thread
* Re: CI slowdown due to PG_TEST_INITDB_EXTRA_OPTS
@ 2026-09-23 18:42 Jelte Fennema-Nio <postgres@jeltef.nl>
parent: Nazir Bilal Yavuz <byavuz81@gmail.com>
0 siblings, 0 replies; 14+ messages in thread
From: Jelte Fennema-Nio @ 2026-09-23 18:42 UTC (permalink / raw)
To: Nazir Bilal Yavuz <byavuz81@gmail.com>; +Cc: Andres Freund <andres@anarazel.de>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-hackers@postgresql.org, Peter Eisentraut <peter.eisentraut@enterprisedb.com>
On Wed, 22 Jul 2026 at 08:56, Nazir Bilal Yavuz <byavuz81@gmail.com> wrote:
> Attached is an attempt to implement 'PG_TEST_SERVER_POSTGRES_OPTS'.
I'm starting to get kinda annoyed that this (or my earlier version)
hasn't been committed yet, or at least made some progress. There's a
trivial way to improve CI runtime by 25-30%. Why has this patch been
in limbo for more than 2 months?
^ permalink raw reply [nested|flat] 14+ messages in thread
end of thread, other threads:[~2026-09-23 18:42 UTC | newest]
Thread overview: 14+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2026-07-18 16:38 CI slowdown due to PG_TEST_INITDB_EXTRA_OPTS Andres Freund <andres@anarazel.de>
2026-07-18 18:47 ` Tom Lane <tgl@sss.pgh.pa.us>
2026-07-20 21:47 ` Jelte Fennema-Nio <postgres@jeltef.nl>
2026-07-21 00:56 ` Andres Freund <andres@anarazel.de>
2026-07-22 06:56 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-09-23 18:42 ` Jelte Fennema-Nio <postgres@jeltef.nl>
2026-08-11 12:39 ` Jelte Fennema-Nio <postgres@jeltef.nl>
2026-08-13 10:09 ` Jakub Wartak <jakub.wartak@enterprisedb.com>
2026-08-13 15:37 ` Greg Sabino Mullane <htamfids@gmail.com>
2026-09-06 18:12 ` Andrey Borodin <x4mmm@yandex-team.ru>
2026-09-07 11:06 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-09-07 13:15 ` Andrey Borodin <x4mmm@yandex-team.ru>
2026-09-10 11:45 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-09-11 06:29 ` Andrey Borodin <x4mmm@yandex-team.ru>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox